SOFTWARE QUALITY ASSURANCE FOR MATHEMATICAL MODELING SYSTEMS

Useofversioncheckingtoolstodeter- mineiftheversionusedinthepreviousofMcialdistribution,thelatestversion intheproductrepository,theversionusedforthebuildintheMakeMleand theversionusedintheauditstringareidentical.Theaccuracyandunisonof versionsisnecessaryforbugtrackingaswell(Seeauditstrings). Productversionsarefrozenautomaticallyforbuilds.<br><br> Thisensuresthat developersdonotaccidentallyoverwriteexistingcodeandinformsthemif versionsneedtobebumped.Becauseofthelargenumberofproductsthat existintheGAMSportfolio,eachwithnumerousversions,automationtools (scripts)fortaskssuchasmakingcurrentproductdirectoriesread-onlyorread- writebecomeincreasinglyimportant. QualityControlandTesting Alltestingactivitiesshouldincludeprocessestouncoverdefectsduringthe completelifecycleofthesoftware.Thefocuson fulllifecycletesting (as opposedtosystemintegrationtestingonly)isimportant,becausethecostof defectsrisesexponentiallythelaterthephaseofthecycle[14].Thetesting activitiesinclude,butarenotlimitedto: Unittesting. Testingoftheindividualcomponent(solverandsolverlink module)usingbothblack-box(input-outputonly)andwhite-box(knownin- ternalcodestructure)typetests.<br><br> Regressiontesting. TestingtodetermineifchangestothesoftwareorMxing adefectcauseanyproblemstoothercomponentsinthesystem. SystemIntegrationTesting.<br><br> Testingdonetoensurethattheentireproduct (basemoduleplussolvermodules)functionsasintendedandtospeciMcations. Theemphasisofalltestingactivitiesisagainonautomationandrepro- ducibility.Manyofthetoolsareavailablepubliclyintheformofmodelli- braries.Themodelsintheselibrariescanbeusedbyjustrunningthemwithin GAMS.In[3],someofthesetoolsandmodelsthatarepubliclyavailableare alreadydescribed. 6 Metrics Mostengineeringprocessesinvolvemeasurementstomakeaccurateassess- mentsoftheattributesoftheproduct.Theuseofmetricsisimportantinthatit quantiMesattributesofagivenprocessoroftheproduct.Inparticular,metrics suchasnumberofdefectsofaparticulartype(critical,serious,cosmetic,etc.) areusedwithinGAMStodetermineiftheproductisreadytomovefrombeta phasetoashippableproduct.Sincedevelopmenttakesplacecontinuously, metricsshouldbecollectedcontinuouslyaswelltodeterminequalityofthe currentproduct.<br><br> 3.SQAForModelingLanguagesandSystems Inthissectionwedescribesomeofthe special problemsofmaintainingsoft- warequalityinthecontextofmodelingsystemswhichdiffersfromtraditional QAprinciples.Furthermore,wegivesomebackgroundonmodelingsystems ingeneraltomotivatetheQAprinciples. BasicTechnicalPrinciples Intheearlydaysofmathematicalprogrammingsystems,theexistingtech- niquestoconstruct,manipulate,andsolvemodelsrequiredseveralmanual, time-consuming,anderror-pronetranslationsintothedifferent,problem-speciMc representationsrequiredbyeachsolutionmethod.Furthermore,thesolution methodswereusuallytiedtoaspeciMcarchitectureandplatformandportabil- ityofmodelswasvirtuallynonexistent. Learningfromtheseearlytechniques,mostmodelinglanguagestoday,in- cludingLINDO[15],GAMS[1],andAMPL[7],appearinginthatchronolog- icalorder,adheretothefollowingbasictechnicalprinciples:(1)separationof modelandsolutionmethods,(2)computingplatformindependence,and(3) multiplesolvers,platforms,andmodeltypes.Theadherencetotheseprinci- pleshasmanyadvantageswhicharedescribed,forexample,in[3].<br><br> DescriptionofComponents Inordertounderstandwherepossibledefectscanoccur,anoverviewof thesystemarchitectureofmodelingsystemsisnecessary.The basemodule (orexecutionsystem)isthecoreofthesystemandisdesignedtotranslate thehuman-levelalgebraicmodelstatementsintothescalarformulationpassed ontothe solvermodules .Thebasemoduleincludesthelanguagecompiler, whichperformssyntaxandothercheckstoensurethe(grammatical)integrity ofthemodelformulation.Ifthecompilationissuccessful,thebasemodule expandsthemodelformulationandgeneratesa(sparse)matrixtobepassed ontothesolvermoduletosolvetheproblem.Additionaluser-speciMedsolver SoftwareQualityAssuranceforMathematicalModelingSystems 7 optionsarepassedontothesolvermodulesaswell.Thesolvermodulesare essentiallyblackboxmoduleswhichreturnsolveandmodelstatusaswellas solutioninformation(ifitexists).Finally,thereexistother externalmodules , whicharenotsolvers,butarelinkedtothebasemoduleinasimilarmanner,to performspeciMctasks.Theseinclude,forexample,linkstoMatlab,Excelor databaseinterfaces,orconversionmodulessuchasCONVERT[3]totranslate themodelintoothermodelinglanguageformatsorstandards. ChanceofFailure Thereliabilityofasystemissometimesreferredtoas"meantimetofail- ure."AlthoughintraditionalSQAterminologyfailureimpliesdefect,inthe MPworldthisreferstoalgorithmicfailuretoMndasolution.Whilephenom- enalprogresshasbeenmadeandwearenowabletosolvemanydifMcultand large-scaleproblems,thesteadystateofmathematicalprogrammingsoftware should conservatively stillbeassumedtobethefailurestate.Unlikeothersoft- waresystems,suchasdatabaseapplications,controlsystems,orothertypesof systems,whereevermoredetailedspeciMcationsanddetailedtestingcancon- tinuallyreducethechanceofdefect(hypotheticallytozero),nosuchparadigm forfailure-freesolvesexistsforoptimization(orothernumerical)software. Inparticular,itislikelythattherewillalwaysexistmodels,whichcannotbe solvedreliably.Indeed,manyofushavelikelyexperiencedunpredictablebe- haviorinmodels,whereachangeofasingleequationordataitemsuddenly provestobetheculpritinfailingtosolve.TheuseofSQAtechniquescan helpminimizethischanceoffailureandprovidegracefulreturninformation ifnosolutionisfound.Itisbecauseofthischanceoffailure,thatcommercial modelingsystemsmustfocusonreliability,inparticularbyprovidinginfor- mativereturninformationincaseoffailure.ThislevelofQAcomplexityisin addition toQAactivitiesassociatedwithtraditionaldefectprevention.<br><br> Defects (asopposedtofailures)canoccuratanyphaseoftheprocessNow andcanresultincatastrophicmalfunctionorundesirablereturninformationif asolutionisnotfound.Thedefectscanoccureitherinthecompilationphase, thedatamanipulationphase,themodelgeneration(matrixgeneration)phase, thesolvephasebythesolvermodules,inthesolvephasebythesub-solver modules(forexample,anMINLPsolvermaymakeuseofanNLPsolver),or duringtheprocessingofthesolutionreturnedbythesolvertothebasemodule. Asolvemayinitiatetheexecutionofothermodelingsystemcomponents.At aminimum,anyrigorousqualitycontroltestingproceduresmustaddressthe possibilityoffailureateachofthesephasesnestedseverallevelsdeep. 8 UniformReturnStatus Becauseoftheinherentchanceoffailure,modelingsystemsmustbeadept atdealingwithvariousreturnstates.Unfortunately,solversarenotuniform intheamountortypeofinformationreturnedtothebasemodule.Atabare minimumthisinformationincludesthesolutionpoint,ifithasbeenfound,but couldalsoincludedualinformationorinfeasibilitiesifthemodelwasfoundto beinfeasible.Inordertoensureuniformreturninformation,GAMSrequires returncodesforboththemodelandthesolverfromwhichonecanaccurately deducetheoverallreturnstate.<br><br> The solverreturnstatus referstothestatusofthesolver,forexamplenor- mal,resource(time)interrupt,iterationinterrupt,nosolution,anderror.For simplicity,numericalcodesfrom1-13areassignedtoeachofthesestates.The modelreturnstatus referstothestateoftheentiremodelorofaparticular point:forexample,optimalsolution,locallyoptimal,integersolution,infeasi- ble,locallyinfeasible,unbounded,orerror.Thesearemappedfrom1-19.The statusoferror(13)shouldideallyneverbereturned.Rather,anyreturnstatus shouldbemappedmorespeciMcallytowell-deMnedreturnstatus.Asdescribed laterinthispaper,theerrorreturncodeistobetreatedasaseriousbuginthe software.Itshouldbeclearthatwithoutuniformreturncodesobtainedfrom eachsolvercall,regardlessofthesolvermodule,itmaybecomeincreasingly difMculttodoconsistenterrorchecking. StatuscodeanalysisishandledinGAMSbyuseof matrix<ltering .Through carefulanalysis,acceptablemodel/solvereturnstatuscodecombinationshave beenpre-determinedandareranked(ahighervalueindicatesabetterreturn statethanalowervalue).Returncodescanbeusedinsystemintegration testingbyusingmatrixMltering:Naggingallmodel/solvercombinationswith model/solverstatuscodecombinationshavingarankingthatislessthanthe speciMedthreshold.Roughly,thehigherthethreshold,thestricterthetesting passcriteria. InTable1,weshowallpossibleacceptablereturncodecombinationswith theirrespectiveranking.Solvestatuscodesareinthecolumnsandmodel statuscodesontherows.ForexampleaModelStatus/SolveStatuscode combinationofoptimalsolution/normalcompletion(1,1)hasahighranking of9,whereasacombinationofErrornoSolution/ErrorSolverFailure(13,13) isneveracceptable(denotedbyadot.).<br><br> 4.SoftwareQualityAssuranceinGAMS GAMSisavailableon7differentplatforms(Windows,Linuxand5other UNIXenvironments)andconsistsofover45differentmodulesandproducts. Thusthebuildprocessisquitecomplexandrequiresasinglestandardreposi- toryfromwhichsourceandlibrariesareextracted.Softwarebuildsforthedif- SoftwareQualityAssuranceforMathematicalModelingSystems 9 Table1. MatrixFiltering.Rankingofmodelandsolverstatusreturncodes ferentplatformstakeplaceondifferentmachines,whichmayexistin-house, orelsewhere.<br><br> Becauseofthecomplexityofportingtoanewmachine,whichrequires multiplecompilersandutilities,initializationofanewmachinerequiresin- stallationofvarioussoftwaremodules(includingspecializedGAMS-speciMc buildscripts).Thisstepisunfortunatelynotautomated,althoughthebuilden- vironmentisrelativelystableandnewportingmachinesareonlyinfrequently introduced.Thebuildprocesscanbedescribedasfollowsforeachplatform: (1)Copygeneralbuildinstructions(intheformofMakeMles)fromthemaster repositorytolocalenvironment,(2)foreachproduct:extractproduct-speciMc buildinstructionsandsourceintheformofMakeMles,buildtheproduct,re- depositbuiltproducttomasterrepository,(3)dopost-buildprocessestocreate installationMles. ThedescriptionofthisprocessisofcoursesimpliMedherein,althoughit shouldbeclearthatwithouttheappropriate buildautomationtools ,thenec- essarystepstodoasinglebuildwouldbeextremelytimeconsumingandthe probabilityforfailurehigh.Furthermore,becauseofdependenciesofsome 10 productsonotherproducts,itbecomesincreasinglydifMculttosortthrough dependenciesmanually. Theportingenvironmentallowsnear-automatedbuildsofentiredistribu- tions.Indeed,asanaddedSQAmeasure,wehavefully automatedthebuild task forWindowsandLinux,sothatafullbuildiscompletedautomatically onceaweek.Suchautomatedfullcompilationactivitiesensuresthatourport- ingenvironmentiscontinuallyina buildablestate .Thisuncoverspotential bugsearlyinthelifecycleandinturnreducescycletimeforfullreleaseof newdistributions.Theautomaticbuildalsoincludesinstallationandtesting activitiessoacontinualanalysisofportingsourceispossible.<br><br> Con5gurationManagement WithintheGAMSportingenvironment,weuseconMgurationmanagement toolstoensurethatproductversionsareconsistentandversionsourceintegrity throughautomaticcodefreezesismaintained.Furthermore,wehaveaconsis- tentauditstringassignmentforeachmodule. AllGAMSmodulescanbeeasilyidentiMedintermsoftheirversionnum- bers,builddate,lastsourcedatechange,aswellassuchinformationforall modulesneededtobuildtheproduct.Sampleauditinformationincludesthe onelineauditstring(hereforCPLEX) Weshouldnotethatmanystandardsourcemanagementandversioncon- troltoolsexist,notablyforexampleCVS( ).Al- thoughwedonotutilizeanyofthese,anySQAactivitiesshouldincluderigor- ousversioncontrolprocessestomanagesource.Ourprocessesmainlyfocus ontheautomationprocessinmaintainingtheintegrityoftheportingreposi- tory.Theargumentforuseofasimplersystemisthatourproductsconsistnot onlyofsourceinasinglelanguage(suchasC++),butcontainsproductswrit- teninvariouslanguages(orconsistsofmoduleswritteninvariouslanguages), libraries,andothertools.Managementofthesevariousproductsandsourceis simpliMedbymaintainingasimpledirectorytreestructurewithsimplemain- tenancescripts.Itshouldalsobenotedthatsomeofthelibraries(orsource codes)areobtainedfromsourcesusingtheirownsourcemanagementsystem. ButanargumentformanagementtoolssuchasCVScancertainlybemade.<br><br> BugTracking AtGAMSweusevariousbugtrackingsystemsinordertocommunicate effectivelywithourexternalsolverdevelopers.In-house,thesystemconsists ofane-mailbasedsystemwhichuserscansubmitbugreportstoandchange statuses.Reportscanalsobeviewedonlineinternally.Becauseoutsidesolver developers(orclients,ifwearedealingwithconsultingprojects)mayhave SoftwareQualityAssuranceforMathematicalModelingSystems 11 Figure1. (Left)Snapshotofweeklybugstatisticsbymodule(inpercent).(Right)GAMS fulltest.Solveaggregationbymodeltype. differentbugtrackingrequirements,wealsoutilizedifferentsystemsfornoti- Mcationfortheseparties.Flexibilityisthekeyandweaccommodatedifferent systemstocoordinatewithexternalcontacts.<br><br> Asanexample,ourin-housebugtrackingsystemsends weeklye-mailbug statistics reportsofopen,closed,andtotalnumberofbugreportsforeach module,whichcanbeusedtodeterminewhenastabledistributionisready. InFigure1(left)weshowasampleweeklystatisticschartofopenandclosed issues(inpercent)foreachproduct. Testing OurSQAactivitiesfocusmainlyontesting,inparticularonautomatedand reproducibletests.InFigure2weshowaNowchartofourgeneraltestingactiv- ities.Thetestscoverthefulllifecycleofdevelopment,buildandintegration.<br><br> Itshouldbenotedthatduringeachtestingphase,potentialnewrelevanttest casesareaddedtothemasterlistoftestmodels,whichensuresthat(1)previ- ousbugswillnotoccuragain,(2)testscanberunautomaticallyduringsystem integrationtesting,and(3)testsarepubliclyavailable. Thismeansthat,forexample,duringunittesting,ifaparticulardefectis found,therelevanttesttouncoverthebugisaddedtothesuiteoftests,sothat duringfullintegrationtesting,thebugwillbeuncovered,ifithasnotbeen Mxed.Suchactivitiesareparticularlyeffectiveinincreasingsoftwarequality fromonebuilditerationtothenextandfosteranenvironmentof continual qualityimprovement .Ourtestcasesareavailableinthefollowing testingli- braries (orsuites),allofwhichareinthepublicdomain. 12 Figure2.<br><br> GAMSSQAtestingactivities. GAMSModelLibrary. Acollectionofover300models,includingprac- ticalmodelsthatcaneasilybeextendedtoproductionmodels.Thecol- lection .coversallsup- portedmodeltypes.<br><br> GAMSTestlibQualityModels. Anewlibraryofmodelsdevelopedfor testingandqualitycontrol.ThesemodelsaredesignedforusebyGAMS staffandsolverdeveloperstodeterminesolvercorrectness,basemodule correctness,specialfunctions,aswellasperformance. GAMSWorldModels.<br><br> Acollectionofreal-worldmodels,aswellasli- brariesfromtheacademicliterature.See . Inordertoprovideuniformaccesstothesetoolsandmodelsforusersof othermodelingsystems,modelscanbeconvertedintoothercommonformats usingtheCONVERTutility(bothasaGAMSandanonlineutility)foruseby non-GAMSusers.See[3]fordetails. SoftwareQualityAssuranceforMathematicalModelingSystems 13 Unittestingisusuallyperformedbythedeveloperonalimitedintegration systemonalocaldevelopmentmachine.Regressiontestingmayinvolvea GAMSinitialtest ,whichsolvesasamplemodelofeachmodeltypewithall availablesolvers.ItalsoincludesthemodelsfromTestlib,whichcheckthe GAMSbasemoduleforpossibledefects.Iftheresultsofthistestingactiv- ityaresatisfactory,thefullsystemintegrationtestisinitiated.Testsarerun aspass/fail,wherepass/failisdeterminedbythethresholdlimitspeciMedfor model/solverstatusreturncodecombinations.SeeSection3(UniformReturn Status).<br><br> Thefullsystemintegrationtest(referredtoas GAMSfulltest )involvessolv- ingalldemoGAMSmodelsfromthemodellibrarywithallrelevantsolvers andsolvercombinations.Fordetails,seetheGAMSLibrarymodel slvtest.gms ( ).Thisincludes atotalofover 16,000totalsolves foreachplatform.Anaggregationsofsolves permodeltypeisshowninFigure1(right). Furthermore,thefulltestalsoincludesrunningthefullsuiteofTestlibqual- itymodels.Thesetestsincludeteststoverifyproperbehaviorforfailure modes.Forexample,wemaytestfordomaininputviolationsofnonlinear functions.Thenumberofqualitytestsrunoneachplatformdiffers,sincenot allsolversareavailableforallplatformsandsomemodulesmaynotexistfor certainplatforms.Forexample,theExcelinterfaceistestedonlyontheWin- dowsplatform.Thetotalnumberofqualitytestmodelsisabout118,withsome containing numerous pass/failtests.Newqualitytestsareaddedcontinuously. Thetestsareruninfullyautomatedmode,usuallyaftereveryfullbuild, withresultssentviae-mailtothepersonbuildingthesystem.Thee-mail includesanypossibleanomalies,aswellasasummaryofmodelandsolve statuscombinations.Ifthereareanomalies,themodelandsolvecombination isrevealed,sothatthebugcaneasilybereproducedonalocalmachine.Thee- mailreportsalsoservethefunctionofarchivingtestresultsforfutureanalysis andcomparison.<br><br> Asampleexcerptofthefulltestsummarysentviae-mailisshownbelow. Theresultsareshownintermsofnumberofsolvesfallingintoeachmodel statusandsolvestatusreturncodes.Theabsenceofamodelstatusof13(Er- rorNoSolution)orsolverstatuscodeof13(ErrorSystemFailure)generally indicatesasuccessfulsystemtest.Inthiscase,athresholdof3hasbeenset, sothatanymodel/solvestatusreturncombinationswitharankinglessthan orequalto3aremarkedasfailures(notshownintheTable).SeeTable1for themodel/solverreturncoderankingmatrix. 14 Figure3.<br><br> PAVERperformanceofCONOPT3withrespecttopreviousversions.(Left)ProMle plots.(Right)Timingcomparisons. MiscellaneousTestsandQAActivities ThetestingactivitiesaboveuncovermanydeMciencies,althoughsomeother issuesmaynotbeaddressedbythesetests.Qualityassuranceisfurtheren- hancedthroughthefollowingproceduresandusingthefollowingtools: NewdevelopmentusingstandardI/Olibraries. Newsolverscanbeattached toGAMSusingstandardFORTRAN,CorDelphiI/Olibraries.Theuse ofstandardlibrariestocommunicatewithGAMSreducesthenumber ofbugsthatareintroduced,increasesreliability,aswellasperformance.<br><br> Thelibrarieshavebeenthoroughlytestedforrobustness,correctnessand performanceovermanyyearsandprovideareliablewaytointroduce newsolvertechnologies. Performance. PerformanceisstillanimportantcriteriainSQA,both forsolverdevelopersandcommercialusers.Benchmarksrunusing largetestsets,canautomaticallybeexaminedusingautomationtools, suchasthepublic-domainPAVERServer[13]toverifysolverperfor- manceandcomparetoothersolvers.Forexample,in[6],performance ofanewversionofCONOPT(CONOPT3)iscomparedtoexistingver- sions(CONOPT1andCONOPT2).TheresultingperformanceproMle SoftwareQualityAssuranceforMathematicalModelingSystems 15 plots[4]createdautomaticallyusingthePAVERServer,veriMedthat CONOPT3indeeddoeshavetremendousperformanceincreasesfrom previousversions.SeetheresultsinFigure4.Theleftsideshows theperformanceproMlesandtherightMguretheresourcetimingsof CONOPT1andCONOPT3.Theuseofsuchtoolsenablesusersto quicklyidentifytrendsoflargedatasets.<br><br> Clientmodels. SinceGAMShasaheavycommercialclientbase,which demandsreliablesoftware,theuseofmodellibrarymodelsmaynot besufMcient.Theirintegratedmodelsmaybeinterfacedwithcom- plexdatabasesandmakeuseofvaryingmodulesthatinteractinspeciMc ways.Thus,GAMSDevelopmenthasstartedaninitiativeofrunning clientmodelsintheirfullinterfacedcapacity.Thisensuresthatmodels willsolveonnewdistributionsastheydidwithpreviousreleases. IndependentSolutionVeri-cation.<br><br> TheEXAMINERtool[8]canbeused forexaminingpointsandmakinganunbiased,independentassessment oftheirmerit.Itisalsousefulwhencomparingthesolutionsreturned bytwodifferentsolvers.Finally,atoolliketheEXAMINERallowsone toexaminesolutionsusingdifferentoptimalitytolerancesandoptimal- itycriteriainawaythatisnotpossiblewhenworkingwiththesolvers directly. 5.ClientModelTesting TheuseofclientmodelsforSQAtestingisimportantbecauseitgenerally involvesmuchmorecomplexinterfacingbetweendifferentmodulesandgives usabetterideaofreal,large-scalecommercialapplications.Italsoprovides auniquemotivation:itgivesclientsassurancethattheiroptimizationapplica- tionsarecompatiblewithnewGAMSsystemreleasesandgivestheexpected results.Byexpectedresults,wemeannotonlytheabilitytosolve,butalsoto Mndthesamesolution 2 andhavesimilarperformanceindoingso.Thelatter isimportantbecauseevenminorchangesinsolvers(forexamplechangesin defaultsettings),mayunexpectedlycausetremendousincreasesinsolvetime, whichmaybeunacceptablefortheparticularapplicationtimeframe.Inthis sectionwedescribeaclientapplication,whichweuseforourin-houseQA clienttesting. MARKAL[9]isoneofthemostwidelyusedenergy/environment/economy planningmodels,playingacentralroleinClimateChangeanalysisfornumer- 2 TodeMnewhatwemeanbyhavingthe 9samesolution 9isoftenanunexpectedlydifMcultproblemanda sourceoffrustrationfortheinnocentuser.Ingeneral,itisimpossibletoexactlyreproducesolutionsto optimizationmodelsbetweendifferentsoftwarereleases,computingplatforms,orreal-timeevents.The term 9samesolution 9needstobedeMnedforeachapplication.<br><br> 16 ouscountriesandcommunitiesaroundtheworld.Developmentiscoordinated byaninternationalgroup,theEnergySystemsAnalysisProgram(ETSAP). Forcurrentinformation,see . Themodelanditsdataaremanagedbyseveraldifferentapplicationenviron- mentsandthemodelcanbeoperatedindifferentmodesofferinganalmost inMnitenumberofpossiblemodelinstances.Oneparticularapplicationenvi- ronment,ANSWER,agraphicalinterfaceisshowninFigure5.Becauseofthe interfacingcapabilities,andthelargenumberofcomponentsintheMARKAL suite,additionalfailuresarepossible,whichmaynotbecoveredbythesimpler modelsinthetestingsuites,describedpreviously.Thesimplerintegrationtest- ingalsodoesnotdoanyperformanceanalysis,whichisoftenvitalinpractical commercialmodelingapplications.Atypicalmodelrunmayinvolvedozensof optimizationstepsandintricatedatamanipulation,allonverylargedatasets.<br><br> Tomakeautomatictestingpossible,substantialrestructuringoftheinternal MARKALmodelmanagementisrequired.Isolationofthecorecomponents andcommunicationwiththeapplicationenvironmentisdoneviatheGAMS DATAExchange(GDX)format.Itisnowpossibletoautomaticallygenerate complexjobstreamsthatcanbeoperatedoutsidetheMARKALapplication environment.Thosejobstreamsarethenusedinroutineapplicationsupport andmayendupinthecustomertestsuite. TheclientmodeltestsetconsistsofanumberspeciMcofjobstreamswith additionalpass/failtestsdesignedandimplementedjointlywiththeclient.The jobstreamsandallrequireddataandsubsystemsarearchivedandbecomepart ofaQAsupportcontract.Theexecutionofsuchtestsuitesisfullyautomated andQAcertiMcatesaregeneratedandarchivedtoallowauditsontheQApro- cess.Exceptionsaretracked,andaGAMSsystemwillnotgointoabeta releaseunlessallexceptionshavebeenresolvedwhich,insomecases,may requirearedeMnitionoftestsorevenmakingchangesintheclientmodels.In amoretraditionalmanufacturingenvironment,onewouldcallthis preventive maintenance. ProprietaryandconMdentialaspectsofdataandsolutionprocessesaddsome additionalcomplications.Insomecasesitmaybenecessarytoencryptcertain partsofthemodelsand/ordatabeforeaclientisreadytoshareajobstream.<br><br> GAMSprovideanumberoftoolstoextractandtransformorhidemodeland datacomponentstomeettheclient 9sneedfornotdisclosingvitalinformation. TheclientmodelssuchastheMARKALtestsuitebecomethemanifestation ofthecommitmenttothecontinuedprocessofqualityassuranceforboththe clientandthesoftwaredeveloper.Thetestsuitesandtheautomatictesting proceduresaresharedbytheclientandthedeveloperandthusdeMneprecisely whatqualitymeansforaspeciMcapplication.Clientmodeltestinghasbecome awinforbothclientanddeveloperandhasbeenmadepossiblebyautomating thetestingprocessandbysharingthetestinstances. SoftwareQualityAssuranceforMathematicalModelingSystems 17 Figure4.<br><br> MARKAL:(Left)Integratedmodeloverview.(Right)ANSWERuserinterface, dataspreadsheetandgraph. 6.Conclusions Wehaveaddressedsomeoftheproceduresnecessaryforimplementingsoft- warequalityassuranceinmathematicalmodelingsystems,showinghowSQA forMPsoftwarediffersfromSQAactivitiesformoretraditionalsoftwaresys- tems.ThekeystepsareautomatedtestingandautomatedconMgurationman- agementtools,whichfostercontinualqualityimprovement.Inparticular,the focusofourtestingactivitiesisonreproduciblefulllifecycletesting.Ourtest- ingframework,whichincludesmodelcollections,datacollectiontools,and dataanalysistools,allowsseamlessfulllifecycletestinginsidetheGAMSsys- tem,withmanycomponentsavailableforoutsideusebynon-GAMScustomers andresearchersingeneral.Wehopethatthesetoolswillillustratetheimpor- tanceofQAactivitiesforacademics,commercialusers,solvervendors,aswell asmodelingsystemdevelopersandhopetheycanbeofbeneMttothemathe- maticalcommunityinimprovingtheirsoftwarequalityassurancemethodolo- gies. Whilesomeoftheproceduresdescribedmayseemdisconnected,thefocus isonreachingthehighestlevelofmaturitydeMnedbytheCMM.MostMP vendors,likeGAMS,operateinaverysmallnichemarketanddonothavethe resourcestofollowconventionalQAprocessesusedinlargeenterprisesthat haverevenuesthousandsoftimeslargerthantheentireMPnichemarket.How canweevolvefromLevel1,theexcitingbutchaoticphase,toLevel5,the maturephase,andbecomeacrediblepartnerforotherindustries?<br><br> Themainthrustshouldbeto:(1)AutomatetheQAprocessandcertiMcation, (2)buildthetoolsintothesoftwareandsharetheQAprocess(3)maketheQA processtransparentandreproducible,and(4)involvetheclients(academicand commercial)andmakethempartoftheQAprocess. 18 References [1]Brooke,A.Kendrick,D.,andMeeraus,A.(1988). GAMS:AUser 9sGuide ,SanFrancisco, CA:TheScientiMcPress.<br><br> [2]AmericanSocietyforQuality.(2004).Onlineathttp://www.asq.org/. [3]Bussieck,M.R.,Drud,A.S.,Meeraus,A.andPruessner,A.(2002).QualityAssurance andGlobalOptimization.C.Bliek,C.Jermann,A.Neumaier,eds. GlobalOptimization andConstraintSatisfaction,FirstInternationalWorkshoponGlobalConstraintOptimiza- tionandConstraintSatisfaction,COCOS2002 ,LNCS2861.SpringerVerlag,Heidelberg Berlin,(223-238).<br><br> [4]Dolan,E.D.andMor « e,J.J.(2002).Benchmarkingoptimizationsoftwarewithperformance proMles, Math.Programming ,91(2)(201-213). [5]Dolan,E.D.andMor « e,J.J.(2000).BenchmarkingoptimizationsoftwarewithCOPS, Tech- nicalReportANL/MCS-TM-246 ,ArgonneNationalLaboratory,Argonne,Illinois. [6]Drud,A.S.(2002).TestingandTuningaNewSolverVersionUsingPerformanceTests, INFORMSSanJose, Sessionon"Benchmarking&PerformanceTestingofOptimization Software." .Seehttp://www.gams.com/presentations/present performance.pdf.<br><br> [7]Fourer,R.andGay,D.M.(1993). AMPL:AModelingLanguageforMathematicalPro- gramming ,RedwoodCity:TheScientiMcPress. [8]GAMSDevelopmentCorporation.(2004).<br><br> GAMS-TheSolverManuals .GAMSDevelop- mentCorporation,Washington,DC:http://www.gams.com/solvers/allsolvers.pdf. [9]Hamilton,L.D.,Goldstein,G.A.etal.(1992).MARCAL-MACRO:AnOverview, BiomedicalandEnvironmentalAssessmentGroup, TechnicalReportBNL-48377 .Analyt- icalSciencesDivision,DepartmentofAppliedScience,BrookhavenNationalLaboratory, AssociatedUniversities. [10]InternationalOrganizationforStandardization.(2004).Onlineathttp://www.iso.org.<br><br> [11]Mittelmann,H.D.(2003).AnIndependentBenchmarkingofSDPandSOCPsolvers. MathematicalProgramming .95,(407-430). [12]Moore,J.W.(1998).<br><br> SoftwareEngineeringStandards:AUser 9sRoadMap .IEEECom- puterSociety,LosAlamitos,CA. [13]PAVERServer(2004).Onlineat . [14]Pressman,R.S.(1997).<br><br> SoftwareEngineering:APractitioner 9sApproach ,4thEdition, Boston,MA:McGraw-Hill. [15]Schrage,L.S.(1991). Lindo-AnOptimizationModelingSystem ,ScientiMcPressseries, Fourthed.,Danvers,MA:BoydandFraser.<br><br> [16]SoftwareEngineeringInstitute.(1994). TheCapabilityMaturityModel:Guidelinesfor ImprovingtheSoftwareProcess .Reading,MA:Addison-Wesley. [17]Shcherbina,O.Neumaier,A.Sam-Haroud,D.Vu,X.-H.andNguyen,T.V.(2003).Bench- markingGlobalOptimizationandConstraintSatisfactionCodes.C.Bliek,C.Jermann, A.Neumaier,eds.<br><br> GlobalOptimizationandConstraintSatisfaction,FirstInternational WorkshoponGlobalConstraintOptimizationandConstraintSatisfaction,COCOS2002, LNCS2861 .HeidelbergBerlin:SpringerVerlag.(223-238).

less