java - Display RTP MJPEG

Question

I'm looking for a solution to display a RTP JPEG stream with JavaFx. I can display jpeg from a file and receive RTP JPEG stream and split it to identify all parameters and data as specify in RFC2435 But I don't know how to convert my JPEG arrays to a displayable Image. I dont want to implement a JPEG Decoder by myself. Any idea?

Answer 1

Leverage JavaFX's built-in jpeg decoder, which should be able to decode the jpeg images in the Image constructor.

class MJPEGViewer extends ImageView {
  MJPEGViewer() {
    // setup a thread which processes the input stream.
    // the processing thread invokes onNewData for each new frame.
  }

  private void onNewData(byte[] jpegData) {
    imageView.set(
      new Image(
        new ByteArrayInputStream(jpegData);
      )
    );
  }
}

The jpegData is a byte array presumed to contain the JFIF data for a frame extracted from the RTP stream.

Executable Sample

This is an mjpeg movie player that plays the movie from: http://inst.eecs.berkeley.edu/~ee122/sp06/ProgAsgns/movie.Mjpeg

Based on the video stream class from Programming Assignment 5: Streaming Video with RTSP and RTP (I do hope this is not your homework assignment class).

According to the video stream class description, it is a "proprietary MJPEG format", so you will need to do your own decoding of your standards compliant format according RFC2435.

The player works, but does have an issue correctly decoding the JPEGs which I have not investigated. Either the JPEGs in the "proprietary MJPEG format" sample movie are not correctly encoded, or the JavaFX JPEG codec has errors decoding frames. The visible artifact is that you can see the image, but the image is not correctly colored (has a pink shade). It is likely an instance of RT-14647 Incorrect display of JPEG images as the pink shade in the video looks the same in the incorrectly decoded JPEGs shown in the bug. You can clearly see the pink shade in the screenshot of the video rendered by the sample code below. The bug only effects some JPEG images (the great majority of JPEG image I have used with JavaFX have displayed fine). So you will just need to try with your video stream to see if the JavaFX jpeg decoder correctly decodes the jpeg images for you.

Rather than replacing the Image in the imageview each time, it is probably more efficient to use a WritableImage and update it's pixel buffer directly, but the brute force replace image method seemed to work OK for me.

import javafx.animation.*;
import javafx.application.Application;
import javafx.event.*;
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.image.*;
import javafx.scene.layout.*;
import javafx.stage.Stage;
import javafx.util.Duration;

import java.io.*;
import java.util.Arrays;

public class MjpegPlayer extends Application {
  public static void main(String[] args) { Application.launch(MjpegPlayer.class); }

  // ADJUST THIS LOCATION TO SET THE LOCATION OF YOUR MOVIE FILE!!
  private static final String MOVIE_FILE = "/Users/lilyshard/dev/playfx/src/fruits/movie.Mjpeg";

  private VideoStream vs;

  @Override public void start(Stage stage) throws Exception {
    vs = new VideoStream(MOVIE_FILE);

    final ImageView viewer   = new ImageView();
    final Timeline  timeline = createTimeline(viewer);

    VBox layout = new VBox(20);
    layout.setStyle("-fx-background-color: cornsilk;");
    layout.setAlignment(Pos.CENTER);
    layout.getChildren().setAll(
      viewer,
      createControls(timeline)
    );

    stage.setScene(new Scene(layout, 400, 400));
    stage.show();

    timeline.play();
  }

  private Timeline createTimeline(final ImageView viewer) {
    final Timeline timeline = new Timeline();
    final byte[] buf = new byte[15000];

    timeline.getKeyFrames().setAll(
      new KeyFrame(Duration.ZERO, new EventHandler<ActionEvent>() {
        @Override public void handle(ActionEvent event) {
          try {
            int len = vs.getnextframe(buf);
            if (len == -1) {
              timeline.stop();
              return;
            }
            viewer.setImage(
              new Image(
                new ByteArrayInputStream(
                  Arrays.copyOf(buf, len)
                )
              )
            );
          } catch (Exception e) {
            e.printStackTrace();
          }
        }
      }),
      new KeyFrame(Duration.seconds(1.0/24))
    );
    timeline.setCycleCount(Timeline.INDEFINITE);

    return timeline;
  }

  private HBox createControls(final Timeline timeline) {
    Button play = new Button("Play");
    play.setOnAction(new EventHandler<ActionEvent>() {
      @Override
      public void handle(ActionEvent event) {
        timeline.play();
      }
    });

    Button pause = new Button("Pause");
    pause.setOnAction(new EventHandler<ActionEvent>() {
      @Override
      public void handle(ActionEvent event) {
        timeline.pause();
      }
    });

    Button restart = new Button("Restart");
    restart.setOnAction(new EventHandler<ActionEvent>() {
      @Override
      public void handle(ActionEvent event) {
        try {
          timeline.stop();
          vs = new VideoStream(MOVIE_FILE);
          timeline.playFromStart();
        } catch (Exception e) {
          e.printStackTrace();
        }
      }
    });

    HBox controls = new HBox(10);
    controls.setAlignment(Pos.CENTER);
    controls.getChildren().setAll(
      play,
      pause,
      restart
    );
    return controls;
  }
}

class VideoStream {

  FileInputStream fis; //video file
  int frame_nb; //current frame nb

  public VideoStream(String filename) throws Exception{

    //init variables
    fis = new FileInputStream(filename);
    frame_nb = 0;
  }

  public int getnextframe(byte[] frame) throws Exception
  {
    int length = 0;
    String length_string;
    byte[] frame_length = new byte[5];

    //read current frame length
    fis.read(frame_length,0,5);

    //transform frame_length to integer
    length_string = new String(frame_length);
    try {
      length = Integer.parseInt(length_string);
    } catch (Exception e) {
      return -1;
    }

    return(fis.read(frame,0,length));
  }
}

Update

I tried running this program again using Java 8u20 early access build 11 on a Windows 7, and the video played back fine without any pink tinge, so I guess that whatever was causing that issue has now been fixed in a later Java build.