Technical Documentation for the MovieMaker^? client/server code

Basic Outline

The MovieMaker^? software is used to send a sequence of images to a server that will generate a movie, and return the movie to the client. Along with the creation of movies, there also exist a few other options, such as retrieving the status or history of the server.

Client/Server Protocol

The following is the communication protocol used by the Client and Server:

1. Client connects to remote server
2. Client sends Action value
3. Server reads Action Value

Now, depending on the desired action:

if Action == MAKE_MOVIE
   1) Client sends the compression quality (int), image width (int), image height (int),
      framerate (int), and the list of TextScreens (LinkedList)
   2) Client enters the following loop:
      a) Wait for the Server to send the required image number (int)
      if 0 < image number < total number of images, then
         b) Retrieve an image from the input directory
         c) Send the file size to the server (int)
         d) Send the byte array of the image to the server (byte[])
      else
         Client transmits NO_MORE_FILES to Server            
   3) Server reads in each successive file by requesting the next image from the client (above loop)
      until NO_MORE_FILES is read from client
   4) Once the Movie is finished, the Server transits a negative number to the Client, which tells
      the client to begin downloading the movie
   5) Once the client gets set to download the movie, the Server transfers 256 bytes of data from 
      the movie to the Client, until EOF is reached.
   6) Server and Client close the connections and perform any necessary cleanup
   
else if Action == PING_PONG
   The following loop is entered:
      1) Client sends a number (int) > 0
      2) Server reads number, if > 0, responds with current number of jobs.
   Once the client sends a number < 0, the loop stops.
   
else if Action == STATUS
   1) The Server responds with a String representation of the Status.
   
else if Action == CLEANUP
   1) The Server attempts to cleanup any temp files, and the response (String) is sent
      to the Client. The response tells how many temp files, if any, were cleaned.
      
else if Action == IS_ALIVE
   1) Server sends a number (int) to the Client, this is just a simple socket test
   
else if Action == HISTORY
   1) Server sends a String representing the last 15 connections to the Client

else if Action == PREVIEW
   1) Client transmits size of image to preview (int)
   2) Client transmits byte array of image data (byte[])
   3) Client transmits the TextScreen to preview
   4) Server renders the new image
   5) Server sends the new image's size to the Client (int)
   6) Server sends the new image's byte array to the Client (byte[])

How the Image to Movie conversion works

The Server uses the Java Media Framework to transcode the series of images to a movie file. This work is largely (infact, its pretty much a direct copy) of the JMF Tutorial's JpegImagesToMovie^?.java example. The differences are the following:

1. The sequence of images now come from the network instead of the disk
2. The images are in PNG format rather than JPEG format
3. The option of either Quicktime+JPEG compression or RAW AVI are now available

Details of a) can be seen in the protocol above, but basically, instead of reading from an image file into the Processor's Buffer, the image is read from the network's ObjectInputStream^?, and put into the Buffer. In regards to supporting PNG instead of JPEG, this mean instead of throwing the JPEG data directly into the Buffer, it must first be converted. A BufferedImage^? is used to store the data, and this data can then be converted easily using the ImageIO^? tools. When making a Quicktime+JPEG movie, this means converting the raw data to JPEG and then putting this data into the buffer as usual. In creating a RAW AVI movie however, we have to throw the raw data into the buffer as a byte array. This requires a quick conversion from an Int array as stored by the BufferedImage^? into a byte array before putting it into the buffer. Also, the AVI spec needs each frame to be time coded.

Class Breakdown

The following represents the order through the different objects involved in transcoding a movie:

Client.java

Connects to the host/port and setups the Action and movie making sequence.

Server.java

Takes the connection request and hands it off to a Connection Object to run in a new thread

Connection.java

Is responsible for the over control of the transcoding. Handles the different Action types and will start the transcoding of a movie.

ImagesToMovie^?.java

Takes the required setup data (movie width/height/fps, streams, and output) and sets up the Java Media Framework end of the transcoding. See comments / JMF tutorials for details.

ImageDataSource^?

(Inner class inside ImagesToMovie^?.java) A simple class that holds a ImageSourceStream^? class. The processor uses this as its transcoding data source.

ImageSourceStream^?.java

Where the real network communication happens. This class's 'read' method gets called by the processor with a Buffer to fill full of data. This data is what is collected off the network and will contain either TextScreen^? image data, or movie image data.

ImageByteArray^?.java

This is used to transport the images over the network by sending byte array data

FileDump^?.java

Once the movie is transcoded, the Client creates a new FileDump^? and attaches its ObjectInputStream^? to it. The FileDump^? then collects the movie from the network and stores it out to disk.

Misc Other Classes:

TextScreen^?.java

Holds the Text and any options for the Text Screen

ServerStatus^?.java

A stand a lone class that can be run from the command line. This is used to get the status, history, etc from the Server

DownloadListener^?.java

A simple interface for allowing an object to be notified of progress in the download of the movie.

RenderProgressListener^?.java

Another simple interface, this allows for listening for changes in the overall render of the movie.

ServerStatusListener^?.java

When ServerStatus^? enters into ping mode, listeners can be attached so that when the server's status changes (from up to down) different clients can be notified.

Important Notes

There are a few portions of the Client/Server/Transcoding code that may not make immediate sense, so I'll point a few out here.

BufferedImages^? aren't serializable, so that is why the byte arrays are sent over the network. Java most likely has a better mechanism for transport, but currently, if it ain't broke...

It would seem the Java Media Framework 'wants' to transcode movies in JPEG format, since it's much more straightforward to do Quicktime movies than it is RGB RAW AVI movies. The movie format is set inside the ImageSourceStream^? class by setting the 'format' variable to what you want. The difficulty arises having to store the data in a byte array. You'll notice in the copyData( BufferedImage^?, Buffer ) method, the int array of data has to be converted to a byte array to be put into the buffer. This was referenced from the web, so a better solution may or may not be available.

Also, in regards to AVI transcoding, it appears that there is some form of bug in the Java Media Framework. Because of this, the program itself is required to time stamp each frame (according to the AVI spec). This is just a simple calculation of the sequence number and the frames per second. This can be found in ImageSourceStream^?.java as well, notably at the beginning of the read(Buffer) method and later in the copyData(BufferedImage^?, Buffer) method.

In case it is confusing, the basic flow of operation with respect to TextScreens^? is this:

If there are TextScreens in the List:
   If we don't have the first image from the client, grab it first and save it.
   If we haven't generated a frame for this TextScreen yet, generate it and save it.
   Calculate the Time to remain on screen for this TextScreen (fps * duration)
   For every read that occurs, decrement this time counter until equal to 0.
   If there is another TextScreen after this one:
      throw out the generated frame, and reuse the saved first image.
      repeat above
   Else
      Add the first frame to the movie, and continue grabbing images from the network.

A Comparator (found in the Client.java) had to be written, since on some platforms (notably, Mac) if the filenames for images were not padded with 0's (so 1.png 2.png 3.png ... ), there would be the issue of, for example, 3.png being sent over the network after 29.png. The comparator searches from the beginning of the string to the first non numerical charactor and will compare those numbers as integers. If the filename doesn't start with a number (say image-1.png), then the strings are just compared as strings.

There are still issues with weird errors (althrough rare) that cause the server to not take the proper course of action. Most, if not all of these errors are from something odd happening in the Jave Media Framework portion. Usually what happens is the server will then sit there thinking everything is going according to plan, but really, its not. This ends up leaving threads with open sockets going, and usually messes up the job counter. Some sort of intelligent timeout system should be devised that will only timeout on errors. Simply setting a timeout timer probably isn't enough, since there are instances where the render job can be going perfectly fine, but might just be going slow (for example, a 30 second timeout works fine for small movies, but if you have one that is even a little bit long, the download process won't ever have a chance to begin, the 30 seconds is up before the server has a chance to finish the render).

-- AndrewRader^?