NNN : Neural Network modelling tool ----------------------------------- U S E R ' S M A N U A L This copy of software is intended for research purposes only All rights reserved. Copyright rests with the authors. version 1.2 (December 1997) --------------------------------------------------------------------- Implementation history 1995-96 (version 0.1, called 'NNN') Design, backpropagation algorithm, graphical display subsystem D.P. Solomatine October 1996 - April 1997 (version 0.2) Converting NNN to Windows interface: Y.B. Dibike, D.P. Solomatine Radial basis algorithm implementation: Y.B. Dibike May 1997 (version 0.3) Updating Windows interface: D.P. Solomatine --------------------------------------------------------------------- NNN is a simple general pupose neural network modelling tool which can run Both in DOS and Windows operating environments. It allows creation of small-scale NNs with one hidden layer. Two types of networks, namely Error Backpropagation and Radial Basis Function networks are implemented in this version. In the case of Error Backpropagation Network all neurons are of sigmoid-type, and the direct input-output link may be also made; while the mapping function of the Radial Basis Network is ofGaussian type for the hiden layer and linear for the output layer. Important feature of this program is that after training, it allows to: (1) run it as an independent executable, for example from another program. To do that, it is activated as with the option /U, or (2) generate Pascal source code that can be then compiled and used as a independent fast and small executable. As an intorductory text on neural networks, the following book may be recommended: On Error Backpropagation Network M. Smith. Neural networks for statistical modelling. Van Nostrand Reyngold, N.Y., 1993. On Radial Basis Function Network M.H. Hassoun, Fundamentals of Artificial Neural Networks. The MIT Press, 1995. These books were used as the initial source of algorithms used in neural networks. NNN CONTROL -------------------------- NNN is controlled thrugh the configuration file NNN.CFG. Typical configuration follows: 1.0 NNN version 15 hids ,the number of neurons in the hidden layer 0.1 (0.1) kappa :by this value weight is increased if the decrease in error is in the same direction as recently 0.5 (0.5) phi :wght is multpd by phi if there is change in sign of error deriv. 0.5 (0.7) theta :momentum - weight on previous val of derivative 0.4 (0.9) mu :weight on momentum 0 direct 1:there is direct in-to-out link 1 ReadWeiYN 1= read weights from WEI file, 0=initialize WEI-file MYTRAIN.PAT < line A > MYTEST.PAT < line B > 0 ISH (InputSHow) nmbr of input displayed as abscissa; 0=to display number of example 1 OSH (OutputSHow) nmbr of output displayed as ordinate 0.05 ExampHitTol : relative tolerance to check if example is hit 1 display? 30 spread constant(sigma) for RBN 1 (BPN)Back Propagation Network ,(1 inable & 0 disable) 0 (RBN)Radial Basis Network , (1 inable & 0 disable) Training set must be placed into the ".PAT" file. First line contains three numbers: number of inputs, number of outputs, number of examples. Each consequtive line contains 1 example (first all input values, then all output values, all in one line). In the current version of NNN - maximum number of inputs: 30 - maximum number of hidden nodes: 30 - maximum number of outputs: 10 (5 in DOS real-mode version) - maximum number of examples: 3000 (2000 in DOS real-mode version) RUNNING DOS version of NNN ------------------------------------------------------------------ TRAINING -------- 1. Set "ReadWeiYN" parameter to 0, then new weight (".WEI") file will be created. 2. Specify training PAT file at . 3. Specify ISH as 0. Then abscissa will be number of example. 4. Specify OSH. This is number of output you want to see. 5. If needed, specify "ExampHitTol". This is relative proximity range which will be used for determining number of hits. Hit occurs when NN generates output vector with all values in this proximity around the observed output vector specified in the PAT training file. 6. Run NNN (without command line options). Training stops after 20000 iterations for BPN or after 30 iterations for RBN, or when Q is pressed. IF YOU PRESSED 'Q', THEN LATER YOU CAN CONTINUE TRAINING BY BPN FROM THE MOMENT YOU STOPPED. CHANGE "ReadWeiYN" parameter IN CFG-FILE TO 1 AND START NNN AGAIN. IF "ReadWeyYN" IS SET TO 0, THEN EVERY TIME YOU START "NNN", THE TRAINING PROCESS WILL START ALL OVER AGAIN. INFORMATION DISPLAYED --------------------- During training the following is displayed: - least square error for output "OSH" - total least square error (these errors are computed on scaled values, and NOT on actual values). - percentage of hits - i.e. percentage of examples for which NN generated output values that are close to what has been observed TESTING (USE FOR MANY EXAMPLES) ------------------------------- Specify test file at , say, MYTEST.PAT. Run NNN /T The results will be placed to file MYTEST.TST PLOTTING -------- Run NNN /P Plot of the current state of network wil be displayed (weights are taken from WEI-file). USE FOR ONE EXAMPLE ------------------- Create file NNN.INP with inputs values (one example). Specify at the name of the PAT-file with which the NN has been trained. Then the correct WEI-file will be used. Run NNN /U Result (vector of outputs) will be in file NNN.OUT. CODE GENERATION FOR BPN ----------------------- Run NNN /C NNN will then use previously created weight file with the name that is specified at in configuration file, but with extension ".WEI". The resulting code will be placed in the file with the same name plus $ in the end, and with the extension ".PAS". For example, if contains filename MYTRAIN.PAT, then after the training the network (running NNN withoput any command line option) the resulting weights will be placed to file MYTRAIN.WEI. Then, if you run NNN /C the generated code will be placed to file MYTRAIN$.PAS. This code can be compiled by any Borland (Turbo) Pascal compiler, or Delphi system, into an executable and used as a standalone program which reads input from NNN.INP and places the output to NNN.OUT. POSSIBLE ERROR MESSAGES ----------------------- Message: Heap overflow error. Solution: Reduce number of examples by changing the first number in PAT-file. Running NNN under Windows -------------------------------------------------------------- The Windows version of NNN is made to provide a graphical user interface where the user can interact with the tool very easyly. It makes it easy to visualise the intermediat results while the network training is in progress. The user can easyly access the configration, pattern and result files while he/she is in the same application environment. It is also possible to see the plots which shows the performance of the network on the training and verification data. This windows version is developed using Delphi Rapid Application Development Environment. The important steps to use this windows application are described as follows. - Run the executable program NNN.EXE from the program manager or startup menu of Windows operating system. You will see the main form where you can interact with the application and see the important training parameters and some of the results during training. - From the main menu in the menubar click OPEN button, and a standard windows dialog box will apear. Select the configration file you would like to use. When you close the dialog box, you will see all the training parameters, the training and verification pattern files specified in the configration file displayed on the main form. - To specify the type of network you want to use, use 'Network type' combo box to select the one you want to use. - To see the architecture of the network choose 'Plot/Network' from the main menu. - To start training the network on the pattern file specified in the configration file, click the 'Train Network' button or choose Network../Train Network' from the main menu; a form which shows a plot of the pattern representationduring training will be dispayed. The corresponding measures of performance (MSE and Percentage in tolerance range) are also displayed on the same form. If you don't want to see this plot click 'Close' button. You can still see the performance indices on the main form. The training will continue till the maximum number of iteration is reached. To stop the training in the mean time, press 'Stop' button. - To see the performance of the trained network on the verification data, choose Plot/Verification plot' or Network../Verify Network' from the main menu. The test result will be placed in a file with extention .TST and the corresponding plot is dispayed. To Save the resulting plot on disc click 'Save Graph' button and to print it click 'Print graph' button on the corresponding form dispayed. - If there are more than one output, to change the output pattern to be desplayed as ordinate in the training or verification plot, choose 'Options/Display type' from the main menu. When a form appears edit the output number to be desplayed in the edit box provided. - To train the network again with different parameters, change the parameters specified in the main form in their corresponding edit fields and run the program once again. - To edit or create the input files containing new data values on which the trained network is to be applied, choose 'Edit/Input File' from the main menu; select or type the input file to be edited from the open dialog box, an editing window with the input file will be displayed. Edit the file and click Save'. - To use the trained and verified network to estimate the output of a new data which the network might not have seen before, choose 'Network../Use Network' from the main menu; select or type the file name where the new input data is kept from the corresponding open dialog box displayed. In similar way specify the file name for the resulting output. By this time the output of the network for the given input will be calculated and the result is diplayed on the screen. (Available only in NNNPro version): - After training and verifying a network, it is possible to generate Pascal source code that can be then compiled and used as an independent fast and small executable. To do this click PASCAL button. The application uses previously created weight file corresponding to the present pattern file and generate a Pascal source code and give it the name of the weight file with an extension ‘.pas'. Then the application display a message whether you would like to see the generated source code; if you select ‘yes' from the message dialog then the code will be displayed in a display window. This code can later be compiles by any Borland (Turbo) Pascal compiler, or Delphi system, into an executable and used as a standalone program which reads input from NNN.INP and places the output to NNN.OUT. - To train the network with a new pattern file which is not in the configration file, choose 'File/Training pattern' and 'File/Verification pattern' from the main menu and select the corresponding files from the dialog box. - To return the setup to its original state (of parameters and pattern files) press 'Reset' button. - To slightly change (randomize) the weights click 'Shake' button. - To Edit the configration or pattern files, choose 'Edit' from the main menu and select the corresponding menu items. Make any appropriate changes in the files and press the save button to effect the changes. - To see the weight file obtained after training or the result file after verification test choose 'View' from the main menu and select the corresponding entry from the menu items. It is possible to see this files but not to edit them. - To exit the the application, choose 'File/Exit' from the main menu. EXAMPLE FILE INCLUDED --------------------- There are two example files included: aw8182.pat used for training, and aw83.pat used for testing the behaviour of the trained network (to test, use /T option). These files represent the data used in the hydrodynamic/hydrologic modelling of a catchemnt in Venezuela. 25 inputs are comprised of runoffs from 21 subcatchments, 3 releases from reservoirs and the water level at the the previous time step. Output is the water level at a certain point downstream. The training file reflects data from years 1981-82, and the test file - from 1983. Reference: D.P. Solomatine, L.A. Avila Torres. Neural Network Approximation of a Hydrodynamic Model in Optimizing Reservoir Operation. Intern. Proc. of the Second Int. Conference in Hydroinformatics, Zurich, September 9-13, 1996. ---------------------------------------------------------------- Please address all questions to: -------------------------------- Dr. Dimitri Solomatine International Institute for Infrastructural, Hydraulic and Environmental Engineering (IHE) Westvest 7 P.O.Box 3015 2601 DA Delft The Netherlands tel. +31-15-2151815 fax. +31-15-2122921 email: sol@ihe.nl (specify subject: NNN) ---------------------------------------------------------------- Visit our Web site on http://www.ihe.nl The author would be grateful for receiving the information about the experience of using the program. Other software tools by D.P. Solomatine: GLOBE : global optimization tool.