How to connect a C/C++ process to a Java application

A question recently came up in the lab on how to connect a Java visualization to a C process, after a bit of thought I discovered that there aren't too many sources that have documented methods so here are a few I came up with that work along with example code.

mkfifo

The first and probably the easiest method on Linux/Unix based machines is to use a FIFO. mkfifo is available on OS X, Linux and probably on Cygwin (I haven't confirmed this). Using a pipe created with mkfifo from both the C and Java standpoint is as easy as opening and closing a regular file. To begin the process we start off by typing

1 mkfifo   pipename

where pipename is the name we would like to give our FIFO.

mkfifo Example:

ERROR

Next we have to set up the C and Java executables, the example program below reads in numbers from ap.txt then averages them in the compiled code and sends the result to the Java VM application to be printed.

C Application Side

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 #include <inttypes.h>
 5 #include <stdint.h>
 6 
 7 /** define input buffer for below **/
 8 #define BUFFER 4096
 9 
10 static const char* exFile = "ap.txt";
11 static char inputBuffer[ BUFFER ];
12 
13 int main(int argc, const char** argv){
14    FILE *fp = fopen(exFile,"r");
15    /**
16     * check and see if the pointer is null in otherwords 
17     * see if the memory location refered to by fp is set...no 
18     * memory location should be zero if you want to reference
19     * it.  Here are some good ways to do this other than the 
20     * way I did it below:
21     *    if(!fp) {do error}
22     *    if(fp == NULL) {do error}
23     * and then there's the way I did it below:
24     */
25    if(fp == 0 )
26    {
27       fprintf(stderr,"Null pointer exception, check file name.\n");
28       exit( EXIT_FAILURE );
29    }
30 
31    //double check and see if an error occured during open
32    const int err = ferror(fp);
33    if(err != 0)
34    {
35       const char* errMessage = strcat("Something bad happened while opening file ",exFile);
36       perror( errMessage );
37       fclose( fp );
38       exit( EXIT_FAILURE );
39    }
40 
41    setbuf(fp,inputBuffer); //set a buffer for input
42 
43    uint64_t num = 0;
44    uint64_t total = 0;
45    uint64_t n = 0;
46 
47    /**
48     *  test for eof, feof(*fp) - returns a boolean 1 (true) if at end of 
49     *  file and  0 (false) otherwise
50     */
51    while( !feof( fp ) )
52    {
53       /**
54        * fscanf returns the number of items it converted using the format
55        * that PRIu64 macro (defined in stdint.h) represents, 
56        * if it's not equal to 1 we don't want to continue
57        */
58       if( fscanf( fp,"%"PRIu64"", &num) != 1 )
59       {
60          /**
61           * you could do a lot of stuff here as far as error 
62           * handling but basically something bad has happened
63           */
64          break; 
65       }
66       total += num; //add to total the value at memory location num
67       n++;
68    }  
69 
70    const double average = (double) total / (double) n;
71    //close the inputfile
72    fclose(fp);
73 
74    //set outfile to FIFO name
75    static const char* outFile = "temp";
76 
77    /**
78     * open FIFO for writing, we'll skip the open check 
79     * since we just opened it externally...for production
80     * code you should check the return of fopen.
81     */
82    FILE *fp_out = fopen(outFile,"w");
83 
84    fprintf(fp_out,"%"PRIu64",%"PRIu64",%f\n",total,n,average);
85 
86    /** 
87     * close output FIFO, this leaves the FIFO but closes the 
88     * file pointer
89     */
90    fclose(fp_out);
91    return( EXIT_SUCCESS );
92 }

Java Application Side

 1 import java.io.BufferedReader;
 2 import java.io.IOException;
 3 import java.io.FileReader;
 4 
 5 public class StreamTest{
 6    private static final String FIFO = "temp";
 7 
 8    public static void main(String[] args){
 9       BufferedReader in = null;
10       try{
11          System.out.println("JAVA SIDE!!");
12          in = new BufferedReader(new FileReader(FIFO));
13          while(in.ready()){
14              System.out.println(in.readLine());
15          }
16          //done, however you can choose to cycle over this line
17          //in this thread or launch another to check for new input
18          in.close();
19       }catch(IOException ex){
20          System.err.println("IO Exception at buffered read!!");
21          System.exit(-1);
22       }
23    }
24 
25 }

The code for the above example can be downloaded here: javaio_fifo.tar.bz2

To run the code, change the javaio_fifo directory then type:

1 make StreamTest
2 make c_app
3 java -cp . StreamTest
4 ./c_app

A more extensive explanation of the Linux mkfifo command can be found in your OS man pages by typing

1 man mkfifo.

When executing, you should see (illustrated below) "JAVA SIDE!!" followed by the total, number count and average from the c process.

Error Loading Image

Pipe with Strings

The second method opens a pipe directly from the C/C++ process using the popen( "target process","w").

Modifications to C Code above

1 static const char outFile = "java StreamTest";
2 
3 FILE *fp_out = popen(outFile,"w";
4 
5 fprintf(fp_out,"Total: %"PRIu64", Integers: %"PRIu64", Average: %.4f\n",total,n,average);

The Java process on the other hand changes to read from stdin input stream.

Modifications to Java Code above:

1 //remove this line
2 //import java.io.FileReader;
3 import java.io.InputStreamReader;
4 in = new BufferedReader(new InputStreamReader(System.in));

The full code base can be downloaded from: javaiopipes.tar.bz2.
As before change directory to the javaio
pipes directory, run the same make options as before only this time no FIFO is created. To run the two applications we only need to invoke the C process by typing:

1 ./c_app

Once the process runs, the message JAVA SIDE: Total: xxx, Integers: xxx, Average: xxx should appear. The complete process is illustrated below.

Error loading pipes example!!

Pipe with Data Stream Method

A third method is a slight modification of the second method. We still use popen but in this case we send data as raw bits instead of ascii encoded strings. Java reads numbers in big-endian format (see what is endianness) whereas C/C++ reads them in little-endian format (at least on x86 architectures) so we have a choice of either converting the numbers going into the pipe first to big-endian or converting them on the Java side once they are pulled out of the pipe. In my code example I've chosen to use the C method htonl() to convert the byte order in C.

C application side:


 1 #include <stdlib.h>
 2 #include <stdio.h>
 3 #include <stdint.h>
 4 #include <inttypes.h>
 5 #include <arpa/inet.h>
 6 #include <math.h>
 7 
 8 
 9 int main(int argc, char **argv )
10 {
11 
12    srand(3);
13    int iterations = 1000;
14    FILE *out_put_log = fopen("c_output_log.csv","w");
15    FILE *fp = popen("java StreamTest","w");
16    if(fp == NULL){
17       fprintf(stderr,"null file pointer\n");
18       exit( EXIT_FAILURE );
19    }
20    const uint64_t size = sizeof(uint32_t);
21    
22    while(iterations > 0){
23       uint32_t a = (uint32_t) rand();
24       uint32_t b = (uint32_t) rand();
25       fprintf(out_put_log,"%"PRIu32",%"PRIu32"\n",a,b);
26       a = htonl( a );
27       b = htonl( b );
28       fwrite(&a,size,1,fp);
29       fwrite(&b,size,1,fp);
30       iterations--;
31    }
32    fflush(fp);
33    fflush(out_put_log);
34    fclose(fp);
35    fclose(out_put_log);
36    return( EXIT_SUCCESS );
37 }

From the code we see that we generate two random unsigned 32-bit values, convert the endianness using htonl() and then write the values to the pipe using fwrite. I've used htonl, however you could just as easily use htobe32, however I've had issues using endian.h on OS X. This operation would be more efficient if I added the two 32-bit values into one 64-bit vector however for the sake of clarity I left the two writes.

Java application side:

 1 import java.util.*;
 2 import java.io.*;
 3 import java.io.PrintWriter;
 4 
 5 
 6 public class StreamTest{
 7       
 8 
 9    public StreamTest(){
10    }
11 
12 
13    /**
14     * readStream - reads a stream from specified param stream, then adds it to array
15     * @param input_filename String
16     * @param length int
17     * @return List
18     */
19    public ArrayList<Data> readStream(int length) throws EOFException,IOException{
20       DataInputStream input_stream = new DataInputStream(new BufferedInputStream(System.in,4096));
21       ArrayList<Data> array = new ArrayList<Data>();
22       Data<Integer> data = new Data<Integer>();
23       while(input_stream.available()>=4){
24          for(int i=0; i<length; i++){
25             data.arr.add(input_stream.readInt());
26          }
27          array.add(data);
28          data = new Data<Integer>();
29       }
30       return array;
31    }
32 
33 
34 
35    public static void main(String[] args) throws FileNotFoundException{
36    
37       ArrayList<Data> array = null;
38       StreamTest t = new StreamTest();
39       try{
40          array = t.readStream(2);
41       }catch(EOFException e){
42          System.err.println("We've reached the EOF signal!!");
43       }catch(IOException io){
44          System.err.println("We've hit an IO Exception!!");
45       }finally{
46          PrintWriter pw = new PrintWriter("java_output_log.csv");
47          for(Data<Integer> d:array){
48             pw.write(d.toString());
49             pw.write("\n");
50          }
51          if(!pw.checkError())
52              pw.close();
53       }
54    }
55 
56 
57    class Data<T>{
58       ArrayList<T> arr;
59 
60       public Data(){
61          arr = new ArrayList<T>();
62       }
63 
64       @Override
65       public String toString(){
66          StringBuilder sb = new StringBuilder();
67          for(int i = 0; i<arr.size(); i++){
68              sb.append(arr.get(i));
69              if(i != arr.size()-1)
70                  sb.append(",");
71          }
72          return sb.toString();
73       }
74    }
75 }

The Java implementation reads in a list of two integers at a time and stores them in an ArrayList data structure. This method can be generalized all Java data types, however care must be taken when sending Java unsigned values since Java only has signed types.

The code for this example can be downloaded from here: javaio_pipes2.tar.bz2.

To run the example change the directory to javaio_pipes2 then type:

1 make c_app
2 make StreamTest
3 ./c_app

Once c_app is executed it will open a pipe using popen() to the Java application then send through pairs of 32-bit values which are then read and stored in an output log "java_output_log.csv" and "c_output_log.csv" to convince doubters that this works you can run

1 diff java_output_log.c c_output_log.csv

and these two should match.

Notes

We've seen that OS level pipes and FIFOs are useful to communicate between the JVM and a compiled (C/C++) process. These principles can easily be applied to other language types (which is why I chose to highlight them). Another (perhaps more performance oriented) way to communicate is through POSIX shared memory, and it follows the same path as the last example. One complication with shared memory is the platform specific location in which you'll open it (e.g., OS X, typically leaves the SHM file handles within the current working directory, whereas Linux will put the SHM handle in /dev/shm by default). If you want to try out the shared memory approach, check out the simplified SHM library from GitHub (https://github.com/jonathan-beard/shm), and integrate it with your project. You'll have to open it from the Java side like a normal file and parse the input. If I get time I'll update this tutorial with a Java FIFO example to make this more concrete. In the interim feel free to drop me a line and suggest improvements.