A question recently came up in the lab on how to connect a Java visualization to a C process, after a bit of thought I discovered that there aren’t too many sources that have documented methods so here are a few I came up with that work along with example code.
The first and probably the easiest method on Linux/Unix based machines is to use a FIFO. mkfifo is available on OS X, Linux and probably on Cygwin (I haven’t confirmed this). Using a pipe created with mkfifo from both the C and Java standpoint is as easy as opening and closing a regular file. To begin the process we start off by typing
1
mkfifo pipename
where pipename is the name we would like to give our FIFO.
<img src=”../img/mkfifo.jpg” width = auto height = auto max-width: 50% alt=”ERROR” />
Next we have to set up the C and Java executables, the example program below reads in numbers from ap.txt then averages them in the compiled code and sends the result to the Java VM application to be printed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>
#include <stdint.h>
/** define input buffer for below **/
#define BUFFER 4096
static const char* exFile = "ap.txt";
static char inputBuffer[ BUFFER ];
int main(int argc, const char** argv){
FILE *fp = fopen(exFile,"r");
/**
* check and see if the pointer is null in otherwords
* see if the memory location refered to by fp is set...no
* memory location should be zero if you want to reference
* it. Here are some good ways to do this other than the
* way I did it below:
* if(!fp) {do error}
* if(fp == NULL) {do error}
* and then there's the way I did it below:
*/
if(fp == 0 )
{
fprintf(stderr,"Null pointer exception, check file name.\n");
exit( EXIT_FAILURE );
}
//double check and see if an error occured during open
const int err = ferror(fp);
if(err != 0)
{
const char* errMessage = strcat("Something bad happened while opening file ",exFile);
perror( errMessage );
fclose( fp );
exit( EXIT_FAILURE );
}
setbuf(fp,inputBuffer); //set a buffer for input
uint64_t num = 0;
uint64_t total = 0;
uint64_t n = 0;
/**
* test for eof, feof(*fp) - returns a boolean 1 (true) if at end of
* file and 0 (false) otherwise
*/
while( !feof( fp ) )
{
/**
* fscanf returns the number of items it converted using the format
* that PRIu64 macro (defined in stdint.h) represents,
* if it's not equal to 1 we don't want to continue
*/
if( fscanf( fp,"%"PRIu64"", &num) != 1 )
{
/**
* you could do a lot of stuff here as far as error
* handling but basically something bad has happened
*/
break;
}
total += num; //add to total the value at memory location num
n++;
}
const double average = (double) total / (double) n;
//close the inputfile
fclose(fp);
//set outfile to FIFO name
static const char* outFile = "temp";
/**
* open FIFO for writing, we'll skip the open check
* since we just opened it externally...for production
* code you should check the return of fopen.
*/
FILE *fp_out = fopen(outFile,"w");
fprintf(fp_out,"%"PRIu64",%"PRIu64",%f\n",total,n,average);
/**
* close output FIFO, this leaves the FIFO but closes the
* file pointer
*/
fclose(fp_out);
return( EXIT_SUCCESS );
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import java.io.BufferedReader;
import java.io.IOException;
import java.io.FileReader;
public class StreamTest{
private static final String FIFO = "temp";
public static void main(String[] args){
BufferedReader in = null;
try{
System.out.println("JAVA SIDE!!");
in = new BufferedReader(new FileReader(FIFO));
while(in.ready()){
System.out.println(in.readLine());
}
//done, however you can choose to cycle over this line
//in this thread or launch another to check for new input
in.close();
}catch(IOException ex){
System.err.println("IO Exception at buffered read!!");
System.exit(-1);
}
}
}
The code for the above example can be downloaded here: javaio_fifo.tar.bz2
To run the code, change the javaio_fifo directory then type:
1
2
3
4
make StreamTest
make c_app
java -cp . StreamTest
./c_app
A more extensive explanation of the Linux mkfifo command can be found in your OS man pages by typing
1
man mkfifo.
When executing, you should see (illustrated below) “JAVA SIDE!!” followed by the total, number count and average from the c process.
<img src=”../img/makeStreamTest.jpg” alt=”Error Loading Image” height=”auto” width=”auto” max-width=50%/>
###Pipe with Strings The second method opens a pipe directly from the C/C++ process using the popen( “target process”,“w”).
####Modifications to C Code above
1
2
3
4
5
static const char outFile = "java StreamTest";
FILE *fp_out = popen(outFile,"w";
fprintf(fp_out,"Total: %"PRIu64", Integers: %"PRIu64", Average: %.4f\n",total,n,average);
The Java process on the other hand changes to read from stdin input stream.
1
2
3
4
//remove this line
//import java.io.FileReader;
import java.io.InputStreamReader;
in = new BufferedReader(new InputStreamReader(System.in));
The full code base can be downloaded from:
javaio_pipes.tar.bz2.
As before change directory to the javaio_pipes directory, run the same make options
as before only this time no FIFO is created. To run the two applications we only need to
invoke the C process by typing:
1
./c_app
Once the process runs, the message JAVA SIDE: Total: xxx, Integers:
xxx, Average: xxx should appear. The complete process is illustrated
below.
<img src=”../img/pipesExample.jpg” alt=”Error loading pipes example!!” height=”auto” width=”auto” max-width=50%/>
A third method is a slight modification of the second method. We still use popen but in this case we send data as raw bits instead of ascii encoded strings. Java reads numbers in big-endian format (see what is endianness) whereas C/C++ reads them in little-endian format (at least on x86 architectures) so we have a choice of either converting the numbers going into the pipe first to big-endian or converting them on the Java side once they are pulled out of the pipe. In my code example I’ve chosen to use the C method htonl() to convert the byte order in C.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>
#include <arpa/inet.h>
#include <math.h>
int main(int argc, char **argv )
{
srand(3);
int iterations = 1000;
FILE *out_put_log = fopen("c_output_log.csv","w");
FILE *fp = popen("java StreamTest","w");
if(fp == NULL){
fprintf(stderr,"null file pointer\n");
exit( EXIT_FAILURE );
}
const uint64_t size = sizeof(uint32_t);
while(iterations > 0){
uint32_t a = (uint32_t) rand();
uint32_t b = (uint32_t) rand();
fprintf(out_put_log,"%"PRIu32",%"PRIu32"\n",a,b);
a = htonl( a );
b = htonl( b );
fwrite(&a,size,1,fp);
fwrite(&b,size,1,fp);
iterations--;
}
fflush(fp);
fflush(out_put_log);
fclose(fp);
fclose(out_put_log);
return( EXIT_SUCCESS );
}
From the code we see that we generate two random unsigned 32-bit values, convert the endianness using htonl() and then write the values to the pipe using fwrite. I’ve used htonl, however you could just as easily use htobe32, however I’ve had issues using endian.h on OS X. This operation would be more efficient if I added the two 32-bit values into one 64-bit vector however for the sake of clarity I left the two writes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
import java.util.*;
import java.io.*;
import java.io.PrintWriter;
public class StreamTest{
public StreamTest(){
}
/**
* readStream - reads a stream from specified param stream, then adds it to array
* @param input_filename String
* @param length int
* @return List
*/
public ArrayList<Data> readStream(int length) throws EOFException,IOException{
DataInputStream input_stream = new DataInputStream(new BufferedInputStream(System.in,4096));
ArrayList<Data> array = new ArrayList<Data>();
Data<Integer> data = new Data<Integer>();
while(input_stream.available()>=4){
for(int i=0; i<length; i++){
data.arr.add(input_stream.readInt());
}
array.add(data);
data = new Data<Integer>();
}
return array;
}
public static void main(String[] args) throws FileNotFoundException{
ArrayList<Data> array = null;
StreamTest t = new StreamTest();
try{
array = t.readStream(2);
}catch(EOFException e){
System.err.println("We've reached the EOF signal!!");
}catch(IOException io){
System.err.println("We've hit an IO Exception!!");
}finally{
PrintWriter pw = new PrintWriter("java_output_log.csv");
for(Data<Integer> d:array){
pw.write(d.toString());
pw.write("\n");
}
if(!pw.checkError())
pw.close();
}
}
class Data<T>{
ArrayList<T> arr;
public Data(){
arr = new ArrayList<T>();
}
@Override
public String toString(){
StringBuilder sb = new StringBuilder();
for(int i = 0; i<arr.size(); i++){
sb.append(arr.get(i));
if(i != arr.size()-1)
sb.append(",");
}
return sb.toString();
}
}
}
The Java implementation reads in a list of two integers at a time and stores them in an ArrayList data structure. This method can be generalized all Java data types, however care must be taken when sending Java unsigned values since Java only has signed types.
The code for this example can be downloaded from here: javaio_pipes2.tar.bz2.
To run the example change the directory to javaio_pipes2 then type:
1
2
3
make c_app
make StreamTest
./c_app
Once c_app is executed it will open a pipe using popen() to the Java application then send through pairs of 32-bit values which are then read and stored in an output log “java_output_log.csv” and “c_output_log.csv” to convince doubters that this works you can run
1
diff java_output_log.c c_output_log.csv
and these two should match.
We’ve seen that OS level pipes and FIFOs are useful to communicate between the JVM and a compiled (C/C++) process. These principles can easily be applied to other language types (which is why I chose to highlight them). Another (perhaps more performance oriented) way to communicate is through POSIX shared memory, and it follows the same path as the last example. One complication with shared memory is the platform specific location in which you’ll open it (e.g., OS X, typically leaves the SHM file handles within the current working directory, whereas Linux will put the SHM handle in /dev/shm by default). If you want to try out the shared memory approach, check out the simplified SHM library from GitHub (https://github.com/jonathan-beard/shm), and integrate it with your project. You’ll have to open it from the Java side like a normal file and parse the input. If I get time I’ll update this tutorial with a Java FIFO example to make this more concrete. In the interim feel free to drop me a line and suggest improvements.