We had a great worksop on October 16th in the University of Chicago.
Paul Girard and Guillaume Plique’s presentation can be found here : http://medialab.github.io/toflit18/chicago/
Loïc Charles and mine is here : presentation-lc
We had a great worksop on October 16th in the University of Chicago.
Paul Girard and Guillaume Plique’s presentation can be found here : http://medialab.github.io/toflit18/chicago/
Loïc Charles and mine is here : presentation-lc
Loïc Charles, Guillaume Daudin, Paul Girard, Silvia Marzagalli, Guillaume Plique will participate to a workshop in the Neubauer Collegium at Chicago University. Here is the program.
From Quantitative to Qualitative Analysis: New Perspectives on Research in Social History
Friday, October 14
9:30 am – 4:45 pm
Neubauer Collegium for Culture and Society
5701 S. Woodlawn Avenue
Chicago, Illinois 60637
This one-day workshop brings together historians and digital humanities specialists who are working building novel kinds of datasets and putting them to a variety of uses. It aims at discussing the creative use of large databases in the context of interdisciplinary projects with a focus on those exploring historical social and economic data in their quantitative and qualitative dimension simultaneously. We will showcase various methods currently in use (or in development) that go beyond traditional
quantitative methods in order to analyse and visualise data in revealing ways.
A conference of the The French Republic and the Plantation Economy: Saint-Domingue, 1794-1803
9:30-10:00 Coffee and Welcome, Paul Cheney (The University of Chicago)
10:00-10:45 Jo Guldi (History, Southern Methodist University), “On the Interpretation of History by Squiggle: Visualizing Change Beyond the N-Gram”; Chair: Elizabeth Chaterjeee (The University of Chicago)
10:45-11:30 Yves Gingras, (History, Université du Quebec à Montreal) “How Bibliometric Methods Can Map the Global Structure and Dynamics of Science from the 17th to the 21st Century”; Chair: Michael Rossi (The University of Chicago)
11:45-12:30 Loïc Charles (Economics, Université de Paris-8) and Guillaume Daudin (Economics, Université de Paris Dauphine), “Mapping the World of Eighteenth-Century Commodities with a Multidimensional Database”; Chair: Paul Cheney (The University of Chicago)
1:45-2:30 Paul Maneuvrier-Hervieu (History, Université de Caen), « Using and publishing a database: the benefits of XML format for an historical research on food riots in the Eighteenth Century »; Chair: Allan Potofsky (University of Paris-7)
2:30-3:15 Silvia Marzagalli (History, Université de Nice), “Visualizing Early Modern Trade: the Navigocorpus”; Chair: Alain Bresson (The University of Chicago)
3:30-4:15 Paul Girard (Médialab, Sciences Po, Paris) and Guillaume Plique (Médialab, Sciences Po, Paris), “Organizing the Reversible Chain of Transformations: from Trade Statistics records to Datascapes”; Chair: Loïc Charles (University of Paris-8)
Cosponsored by the Franke Institute for the Humanities and the France Chicago Center.
We had a great workshop on September 16th. The program and all the presentations are available here: https://groupes.renater.fr/sympa/d_read/hist_trade_stat/TOFLIT18_2016_September_Paris/. You can see below a quick report on the meeting,
“Classifying international trade before 1945”
Location: Sciences Po (Paris)
Marc Badia, Leslie Bermont, Bertrand Blancheton, Ana Carreras Marin, Léo Charles, Loïc Charles, Guillaume Daudin, Béatrice Dedinger Michel Fouquin, Pierre Gervais, Paul Girard, Michael Huberman, Jules Hugot, Wolf-Fabian Hungerland, Ragnhild Hutchison, David Jacks, Aidan Kane, Thomas Le Roux, Ivan Ledezma, Pierrick Pourchasse and Jan Willem Veluwenkamp and several students.
Short welcome introduction by Guillaume Daudin (local organizer)
Making historical international trade data useful (Germany, 1880-1913)
Presentation by Wolf-Fabian Hungerland (Humboldt-Universitat zu Berlin)
Wolf Fabian discusses the use of the German source and how the goods therein can be classified. Source volumes are disparate: different number of year per volume, number of categories and classification different. This is what it is very useful to use a SITC classification.
Then, he opens a discussion on the matching issue (how to find a SITC equivalent to the original product as it was labeled in the source volumes), utilization of SITC 4 was privileged. Results are encouraging so far.
Object of the database: study the extensive margin over the long period.
Standardization of Latin American trade statistics. Sources and methodology for a study on the diversification of export baskets
Presentation by Ana Carreras Marin (Universidad de Barcelona).
Ana Carreras discusses the industrial and geographical diversification of Latin America’s trade c. 1913. There is Trade-off between standardization and the specificity of the economy one is analyzing.
Research issue : multi-goods partners in extensive / intensive margins.
There is a discussion around whether it is a good idea to use the SITC classification?
Domestic Barriers to International Trade: New Evidence for Brazil, 1920‐1940
by Michael Huberman (Université de Montréal)
In his paper, M. Huberman addresses the issue of the relationship between coastal domestic trade and international trade. Was the former a substitute for the last, or was it part of an educational process of the entrepreneurs that subsequently became exporter ?
Research issue : the study domestic trade in parallel with international trade.
Custom 15: Constructing a database of Ireland’s international trade 1698-1784
Presented by Aidan Kane
Data capture and cleaning of trade data for each year 1698-1784 is now essentially done, and will be soon generally available. Data typically distinguishes six or seven major trading areas. The data are available for around 25 Irish ports, in each year, and the originals are remarkably consistent in format over the time period. Aidan underlines the small number of commodities (500s). 140 volumes of ledgers digitized over the period 1698-1784. Exports much less precise than imports
They did the arithmetic checks automatically
Right now, no classification but he is looking forward to use SITC.
English/British/UK Trade Statistics 1700-1899
Presentation by David Jacks (Simon Fraser University and NBER)
1 out of 10 years digitized. Around 130,000 raw observations (100,000 in SITC). Looking at gravity (distance versus empire)
Problem of prices (resolved at an aggregated level by indexes)
Names of commodities normalized into the database (original spelling not retrievable)
Choosing a Product Nomenclature: The French Case 1836-1938
By Bertrand Blancheton
Working on « commerce spécial » (without transit trade)
The impact of tarifs on specialization ? Choice of classification : SITC versus ISIC ?
The Norwegian customs project- classification and standardization – a practical solution?
By Ragnhild Hutchinson (University of Oslo and Norwegian Institute of Local history)
27 ports for 9 years from 1686 to 1835. The project tried 4-digit SITC and the 1870 classification, but it did not work out : too much time and the match was no good (classification too modern).
This no classification so far, but she wants to implement a suggested classification option as in Navigocorpus.
Funding obtained mostly through private donors, but it constrained the project somewhat: short-term funding renewed several times and for teaching rather than research purpose.
Sound Toll Registers Online, 1497-1857. Standardisation and classification issues.
By Jan Willem Veluwenkamp (Groningen)
Again, the difficulty of using a precise SITC category is apparent.
Your ‘goods’ are not my ‘products’; a preliminary methodological discussion of our instinctive belief in the permanence of things over time and of its unintended consequence
by Pierre Gervais (Paris 3 Sorbonne Nouvelle)
Argue that the main difficulty that arise with any classification of the trade of traditional economies is that it had to cope with commodities that are to a large extent unique for each local context (an ‘indienne’ in Bordeaux in 1726 might not be the same an ‘indienne’ in Marseille in 1726 or an ‘indienne’ in Bordeaux in 1756, etc.)
Wrap-up session :
Guillaume Daudin presented the datascape methodology and how it was applied in the TOFLIT18 project to manage the French balance of trade dataset. Then, he moved on to show some preliminaty results and how the visualization of the relation between commodities illustrates how La Rochelle changed from a port with a diversified set of exports in the 1720s to a much more specialised one in the 1780s.
Loïc Charles gave some concluding remarks. Suggest that they are two different strategies to build a trade database : either to digitize the data and try to save up as much as you can or try to simplify. The latter was the only option available before the IT revolution of the 2000s. But now, it is of the utmost importance of keeping everything in the database.
Second point : You do not have to define a classification ex-ante. Quite the contrary, the philosophy of the datascape is to have a tool that allow the ex-post creation of several classifications in a relatively short period of time (a few weeks). The argument is that each research issue requires more or less a different classification : if one wants to work on an historical issue, it is better to use a classification close to the sources, while SITC is better for long period studies especially when comparison with 20th century is wanted.
Finally opens a discussion on a possible convergence at a European level, through an ERC project.
J’ai le plaisir le vous informer de la tenue d’une journée de travail « Classifier le commerce international avant 1945 » à SciencesPo Paris (salle Goguel) le 16 septembre 2016.
Vous êtes tous cordialement invités. Pour des raisons de sécurité (et pour que nous sachions combien de plateaux-repas commander), merci de vous inscrire auprès de Biljana Jankovic firstname.lastname@example.org.
September 16th 2016
Salle Goguel, SciencesPo, 27 rue Saint Guillaume, 75007 Paris
The long history of globalization is blessed with a comparative abundance of data. Foreign trade has long been a sizable source of income for states and the latter have been careful in collecting information on it: series on international trade starting in the eighteenth century exist for several countries and regions such as England, France, Sweden, the Austrian Netherlands, Venice, Portugal, Spain. (Charles et Daudin 2015). The Sound records go back even further. Hence, much information on past economies may be gained through the systematic collection of these data. This calls for the creation of large databases and there are a number of active projects, under different states of readiness: Ireland, Norway, UK, World, France, Germany, the Sound… The identification and classification of commodities is at the core of any project that want to use these databases for comparative analysis or even to simply use the information collected to its potential.
For example, in the French case, there are more than 45,000 different merchandises mentioned in the 18th century sources (for 400,000 trade flows). Correcting for mistakes made by the original scribes and the person in charge of the transcription brings that number down to 20,000. Further identifying synonyms brings it down to 16,000. It is however necessary to devise some sort of aggregation process to be able to use the data collected to answer a wider set of economic history issues such as those linked to the question of trade diversification, intra-industry trade and revealed comparative advantages.
An additional difficulty is international comparison. There are more problems linked to it than the sole issue of translation. The creation of internationally comparable merchandise classification was a long process starting only in the late 19th century (Kolesnikoff 1953) and it is still a work in progress. Yet, it is central to any comparison process.
Choosing a classification to categorize commodities should meet several criteria: it has to be informed by past practices of recording flows (which are different from one place and one time to another), the characteristics of the past society/economy where trade took place, convenience (as classifying goods is hard work), and the research issue. Among the possible options are: 1. the adaptation of modern nomenclatures, which have a great level of detail, accuracy and comparability, but hardly relates to the economic issues of past societies; or 2. the creation of a tailor-made historical nomenclature, which takes into account the historical context of commercial exchange but begs the issue of intertemporal (and often geographical) comparisons. Moreover, creating a nomenclature takes a lot of time in particular if one wants to produce disaggregated categories. A flexible system of classification that can be integrated in data exploration in order to be able to create detailed categorization for specific purposes within a reasonable amount of time and resources would be ideal. But that requires to invest a substantial amount of work in the creation of sophisticated digital tools, as endeavored by the project TOFLIT18 (http://toflit18.hypotheses.org/).
Wolf-Fabian Hungerland (Humboldt) «Making historical international trade data useful: German product-level data from 1880 to 1913 and the Standard International Trade Classification (SITC)» .Co-author : Chris Altmeppen
Ana Carreras Marin (Barcelona) «Standardization of Latin American trade statistics. Sources and methodology for a study on the diversification of export baskets » Co-authors: M. Badia-Miró and A. Rayes
Michael Huberman (Montréal) Domestic and International Trade in Interwar Brazil: Complements or Substitutes?
Aidan Kane (Galway) « Customs 15: constructing a database of Ireland’s international trade 1698-1784 »
David Jacks (Simon Fraser) « British Trade Statistics, 1700-1899 »
Bertrand Blancheton (Bordeaux) « Choosing a product nomenclature: the French case 1836-1938 »
Ragnhild Hutchinson (University of Oslo and Norwegian Institute of Local history ) « The Norwegian customs project- classification and standardization- a practical solution? »
Jan Willem Veluwenkamp (Groningen) « Sound Toll Registers Online, 1497-1857. Standardisation and classification issues. »
Pierre Gervais (Paris-3) « your ‘goods’ are not my ‘products’; a preliminary methodological discussion of our instinctive belief in the permanence of things over time and of its unintended consequence. »
Loïc Charles (Paris-8) and Guillaume Daudin (Dauphine) : «Multi dimensional classification of goods in a datascape : the TOFLIT18 project»
General discussion and wrapping up
This event is open to all. It is funded by the ANR TOFLIT18 (http://toflit18.hypotheses.org/)
Organizers: Guillaume Daudin (email@example.com) & Loïc Charles (firstname.lastname@example.org).
Inscriptions & information: Biljana Jankovic (email@example.com) 01 45 49 72 51 or http://www.sciencespo.fr/evenements/#/?lang=fr&id=5027
Charles, Loïc, et Guillaume Daudin, éd. 2015. « Eighteenth Century International Trade Statistics: Sources and Methods ». Revue de l’OFCE (Special Issue), no 140: 7‑377.
Kolesnikoff, Vladimir S. 1953. « Commodity classification ». In International Trade Statistics, par R. G. D. Allen et J. Edwards Ely. New-York and London: John Wiley & Sons Inc. and Chapman & Hall, Limited.
July 5th 2016 meeting
Attending: Loïc Charles, Guillaume Daudin, Béatrice Dedinger, Frederico d’Onofrio, Meng Ying Du, Pierre Hollegien, Pierre Gervais , Paul Girard, Corentin Ponton, Grégory Tible, Quentin Vidal.,
Following the meeting with Federico and Tena, some changes have been implemented to the datascape: bilateral trade flows and world flows are now separated; the Federico-Tena database has been added in the metadata view where partners can be displayed by color. Moreover, it is now possible to compare the Ricardo’s best-guess with the Federico-Tena’s one in the country view: they largely match except for a period where exports are missing.
The project is to be presented at the digital humanity conference newt week.
Hasna achieved her tasks: she completed every import in Marseille, and the ones in Nantes until 1761. Requia has not made such good time: about 10 years of exports in Marseille are still to be retranscribed (1741 and 1742 are very difficult to read and are hard to read, Loic will have to deal with them).
Demba is going to complete Bordeaux and will start transcription for imports or export in Nantes.
Loïc is still checking transcriptions. There are many difficulties: typos (easy to correct), computing errors, wrong unity of measure (more annoying), etc. Trade records were on average better done in big direction than in small ones. Marseille represents a lot of works (about 2500 lines per year – a big day of work is not enough to deal with one year).
Total sums computed by the copyist does not always match with the ones computed by ourselves, but the later are to be preferred because of the excessive rounding or computing errors made by copyists.
Since 1815, merchandises’ names appear to be much more heterogenous, which increases the work of orthographic normalization and simplification. However, it can be explained by the large number of braces in the products’ labels that are used to distinguish classification and specification.
Pierre Gervais is pursuing the classification of the goods, but his facing some difficulties due to the state of the “simplification” file. Most problems wil be tackled by an algorithm that does not integrate every spelling yet.
Beatrice proposes to make a unique and universal classification of products from the 18th century. It sounds like an enormous project for the moment though
New classifications have been added up to the datascape or updated: Medicinal Products by Pierre Hollegien, North American products by Quentin Vidal, Eden Treaty by Corentin Ponton. The grains’s classification made by Federico d’Onofrio is still to be uploaded. “Revised” classifications are now the standard ones.
Bugs of interface are being fixed, especially when selecting parameters for plotting network views.
Guillaude Daudin asked Paul Girard and Gregory Tible on the possibility to implement a new feature in the datascape. It would consist in a tool allowing us to pick terms from a daughter classification and use them in a mother classification. The mother environment would be filtered by the daughter classification.
No more TOFLIT meetings before the end of the summer holidays.
Research of a room that could host the 16th September meeting, and discussion about an accommodation in Chicago for October.
Merci à Corentin et Quentin pour la première version du compte rendu de la réunion du 8 juin.
Guillaume Daudin, Béatrice Dedinger, Meng Ying Du, Pierre Gervais , Paul Girard, Corentin Ponton, Grégory Tible, Elisa Tirindelli, Quentin Vidal.
Imports coming from Marseille are done, but are still to verify.
Marseille’s exports however should be done in two weeks, as well as 1789 sources.
Moreover, Demba will achieve entering Nantes and Bordeaux sources before long
Cyril Canet is doing Rennes and Nantes ones
Quentin Vidal is completing the 1823 Table of Quantities, while Corentin Ponton in entering unit prices from 1819 to 1821.
Since character recognition does not work (even on printed registers), manual retranscription is necessary, even though it is long and dull. Anyway, we really would like to stop it by the end of summer, so as to focus on the database exploitation.
1789 is nearly done (only 4 directions are left) 1789, and then it will be over.
For England, 1787 and 1788 still need to be done.
Idem for Marseille’s imports. It is a long job (between 2000 and 3000 observations each year).
3) Identification of commodities and classification
We are still following the 21th century Standard International Trade Classification (SITC) proposed by United Nations to group the nearly 20 000 goods of the dataset by category. However, Pierre Gervais is facing some difficulties regarding particular products and groups which are completely meaningful nowadays but did not make any sense back in the 18th century.
For instance, he evokes the case of medicinal products that were often considered as food supplies in those days (such as the saffron, which is also used as a dye). Besides, dye plants and tinctures are also mistaken.
Tea is an issue for 0a/0b as it can be both be produced by local farmers or on plantations.
So far, about 20% of the 15 000 goods are classified, although a large part of them goes in “category x” whenever a doubt arises. By the way, a great share of commodities is also labelled as raw materials, which is certainly overestimated. This work should be done by the end of the month.
Furthermore, Elisa’s Hambourg classification is available on the datascape, while Quentin aims at creating a classification for specific Canadian products and Corentin for products impacted by the Eden Treaty
4) Elisa Tirindelli
Elisa Tirindelli defended her master thesis this week: she tried to explain the effects of war on French exports by distinguishing the effect on colonial exports (sugar and coffee) and non-colonial ones (wine and spirits).
She will pursue a PhD track in Dublin next year in historical and urban economics.
Paul Girard and Gregory Trible have improved the performances of the website. They created two new tabs from the “Global tab”: Countries Network and Product Terms. The former is not completely working by now but will display interesting information on countries relationships in trade, while the later could return details on the complexity of merchandises by analyzing every term of each commodity.
The metadata tab has seen a few improvements. However, we would also like to aggregate flows by classification. For instance, we would like to see how the “pays grouping” classification has evolved through the decades so as to understand how Toscane, Piémont and other provinces were assimilated in Italy by the statistics.
Besides, as some particular mixes of criteria do not obviously exhibit any results (since they are completely irrelevant), we should give users more information and advises regarding requests, by the means of help pop-ups or FAQ.
Gregory Tible’s internship is nearly over; he will leave Sciences Po at the end of July. Paul will Guillaume Plique might punctually intervene on the datascape. But Paul will do the main work. The additional budget will be necessary to find some help.
Ricardo is doing well. The team has been chasing bugs and developing the mew Metadata view (issue of mirror rates)
After that, it will be back to data management (eg source cleaning).
7) Next meetings
• June 22nd: (Ricardo meeting) with Giovani Federico and Antonio Tena
• July 5th, 9.30am at the Médialab: last TOFLIT meeting before summer holidays.
• September 16th – 17th: public meeting with people from the English project, Norway, Ireland and the STRO (7 guests have already favorably answered); internal meeting on Saturday 17th.
• October 14th in Chicago: need to obtain more details from Loic Charles.
May 11th Toflit meeting
Attending : Requia Bouali, Loïc Charles, Guillaume Daudin, Béatrice Dedinger, Federic d’Onofrio, Meng Ying Du, Hasna El-Adel, Pierre Gervais, Paul Girard, Corentin Ponton, Grégory Tible, Elisa Tirindelli, Quentin Vidal
– For Marseille, Requia is doing year 1749 and Hasna 1751.
– Laura is working on 1789.
– By the end of May, we should have most of Marseille + Nantes.
– Demba will finish afterward. We plan to stop in July.
– Corentin Ponton adding some unit values from 1815 to 1821. But they are not bilateral : it seems like there is a double aggregation of values and quantities because computing of average prices exhibits wrong results.
– Quentin Vidal is transcribing post-1815 sources by the customs, that indicate quantities and duties.
• Loic Charles is checking the retranscription of Grenoble and England sources (4 to 5 thousands lines a day). He expects to end this job by September.
After summer holidays, we should stop retranscripting of new sources so as to focus on the database exploitation.
– Identification of commodities is nearly done (thanks to Quentin and Corentin); some doubts remain concerning strange words, though most of them might come from foreign languages (Dutch or Flemish mainly). Note that those unknown goods are worth only small amounts.
– We could use Beccia’s information to enhance the identification
– We could identify some Indian textiles because they all end the same way.
– There are copyist mistakes that are very difficult to correct.
Pierre Gervais and Guillaume are working on the SITC variant we will be offering as one of the classifications. We discuss the difficulties.
• SITC: consists of a dozen of categories, divided into more sub-categories.
o Motivation: building bridges between different projects and between different centuries.
o Drawbacks: frontier between categories might be too blurred, and some goods create ambiguities, about the transformation degree for example. There are way too much commodities in the category “9C: other products”.
o Should we create a new version of the SITC classification, maybe more fitted to our project but less standardized and thus less internationally recognized?
• Examples of problems encountered during the classification: What to do with some chemical products, such as medicines that are not always transformed products? With textile goods, made of cotton, silk or something else we do not know?
• Ideally, we aim for a classification perfectly understandable for someone who does not know anything about the project, or at least anything but the rules of the classification.
6. Quantities in the datascape
We discuss how to integrate quantities in the database. Guillaume suggests we could create one variable / facet for each type of quantities (weight, volume, number and maybe length). In that way, they could be treated like values in the datascape.
7. Work on the datascape
We have a quick discussion on progress of the datascape, and how to treat 1749 and 1751 for National Best Guess
We admire the new version of the Metadata view.
9. Next meetings
• June 8th 2016 (Toflit meeting), at 9h30, Medialab.
• June 22nd 2016 (Ricardo meeting) with Giovani Federico and Antonio Tena
• July 5th: last meeting before summer holidays.
• September 16th : public meeting with people from the English project, Norway, Ireland and the STRO. There will be an internal meeting on Saturday 17th.
• October 14th in Chicago (financed up to $4500 from Chicago side). It would be nice if Pierre, Paul and Siliva could come.
Réunion Toflit 13/04/16
Requia Bouali, Marie-Laure Dagieu, Guillaume Daudin, Béatrice Dedinger, Hasna El-Adel, Paul Girard, Laura Mouton, Grégory Tible, Elisa Tirindelli, Du Meng Ying,
The project is presented to out two new interns : Hasna El-Adel and Requia Bouali. They are working on the retranscription of Marseille and Nantes. We discuss how to use the datascape to facilitate the retranscription
INED has 35,000 € left. We expect to spend approximately 20,000 € this year. At the end of 2016, 80% of the initial budget will have been spent, mainly on wages and contractors.
Sciences Po has 25,000 € left at the end of 2015. 60% of the initial budget has been spent, mainly on wages. After Guillaume Plique and Gregory have been paid, we will have approximately 10,000 € left. We discuss whether we want to transfer this money for other workers in 2017.
Dauphine has 70,000 € left. Only 15% of the initial budget has been spent. This is because Dauphine will mainly take in charge missions and conferences, which will be more important at the end of the project when we will be able to present the data and our research results. We are late in the collection and organization of the data.
Pierre Gervais is currently working on the SITC classification.
We discuss the chronological repartition of good names. Only roughly 10% of products are mentioned every year, suggesting that semantic variations are very important.
Paul Girard is working on making updates to the datascape easier so that they can be done regularly.
Gregory is working on improving the interface and squashing bugs. For exemple, the server breaks down under searches crossing countries and goods in the indicator visualization.
There is still a bug when trying to search by crossing simplification pays and direction.
The source names have been nearly fully streamlined.
Du Meng Ying is working on a new « metadata» visualization. It shows which data are available from which place: they can be sorted by number of flows in the whole period or by average value of flows.
We discuss the use of absolute value of trade or share for sorting purposes. It would be better to calculate average value as average of the share of what is available, to avoid countries with trade only in 20th century to stand out just in value terms.
Each main viz will have a metadata equivalent. This is done for the Country and the World view, but not yet for the Bilateral view.
Next meeting is May 11th at 4pm at the médialab 13 rue de l’Université. We discuss our schedule for the end of June without reaching a definite solution.
Thank you to Elisa who wrote the first version of this report.
Guillaume Daudin, Béatrice Dedinger, Pierre Gervais, Paul Girard, Guillaume Plique, Grégory Tible, Elisa Maria Tirindelli, Du Meng Ying
We have now a new version of normalisation orthographique. We discuss the integration of the work of GD, GP, PG, and ET.
We also have a new version of «simplification».
GD will be working on a new SITC before Cambridge.
We know have a more-or-less working datascape at :
If you want to explore it, please ask Guillaume Daudin for access codes.
The paper was accepted and Ricardo will be presented in July. Du Meng Ying is doing her internship on Ricardo at present.
Major work to be done on Ricardo:
– Correcting bugs and adding new views (like the metadata in toflit).
– Creating a flow of data, converging to what was done in toilet. Transferring everything in a github account.
The breakdown into sources is still to be done.
We discuss the integration of an additional « world» trade estimate.
3) Merchandise treatment
– Normalisation orthographique is nearly done. It classifies 44,000 items into 21,000 groups. The only data missing are the data that have not been acquired yet. According to Loic there are 25% more lines to be done, even more according to Pierre.
– Simplification brought normalisation down to 16,500 lines. It still needs to be revised in detail because it still contains some old version of the work. Pierre suggests to organise a group meeting to talk about it and to work on it in a more organized way.
– Original orthographique needs to be cleaned. some products are still unknown and it is necessary to understand what they are. Pierre says they are not many.
– Catégorisation issue:
Pierre G. is of the opinion that the 1789 categorization (c. 250 categories) is not very coherent and might not work well on the entire dataset.
Guillaume D. suggested to use categorization 1788 because it contains all the data and in addition it links XVIII to XIX century data.
Pierre is skeptical, in his opinion we are looking for categories that will be useful to economists and economic analysis and maybe the best idea would be to come up with a new proposal of classification in the style of 2 digit sitc.
In the end, we decide to work directly on a 20 groups categorization (sitc18_rev3). GD is to do the job (he needs it for Cambridge !)
4) The June meeting in Chicago project is cancelled. We will try to do an other in December or late winter. We will try to re-use part of our ideas for the 2018 Boston WEHC conference ?
5) Schedule for next meetings :
– 13 April 13 14.30, room 84 rue de Grenelle
– 11 May 16.00, medialab
– 8 June 9.30, medialab
– 1 July 9.30, medialab
6) Paul suggests to do a “for paper sprint” in July (1st ?) or September to provide data generation and exploration specifically need to paper in progress. We need to discuss that.
7) 2018 Boston WEHC conference : do a BYOD (we need to contact organizers). Or maybe a big database thing with Berlin ?
Meeting Toflit18 – 19/02/2016
Loïc Charles, Guillaume Daudin, Pierre Gervais, Guillaume Plique, Grégory Tibre, Elisa Tirindelli
!!!!! Change of schedule !!!!!! Next meeting will be on Wednesday March 16th, at 14.30 at the médialab.
1. New data
Lyon has given a dilatory answer. Dunkerque would require one week of work. So we will not pursue these opportunities, focus on what we have already and work on it.
There should be 10 «easy» years to collect each in Narbonne, Annecy and the bibliothèque de l’institut (Paris). One would need te be sponsored for last one. Loïc will give quote for these : we might send interns there.
No news concerning Bayonne.
1789 imports are done.
Marseille sill needs to be done + 2 two years for Nantes + the Eden treaty data.
1789 imports are done
Laura is working on 1789 exports.
Marseilles is finished (Thasectra + 2 interns).
Cyril or Dauphine interns must still do Bordeaux, and the remaining years for Nantes and Rennes.
– Elisa Tirindelli (PSE M2 student) will work till May 31st/early June. She will work on trade with the North for her master thesis + on simplification.
– Corentin Ponton (Dauphine M1 student) who will work on the project from May 2 to July 29. He will contribute on unknown products. His aim is to build a gravity model.
– Quentin Vidal (TSE M1 student) will work from April 18 to July 13.
– Victoria Sessego (Cachan 1st year student) might start working in September one day a week).
– Requia Bouali (Paris-8 AES + M1 Sociology + M2 Anthropology) will work from April 4 till July 1 every day except for Tuesdays on retranscriptions (till May 15th) + a thesis on the organisation of INED
– Hasna El Adel is a Paris-8 M2 student (with a licence in Economics). She will work on retranscription (till May 15th) + a thesis on the TOFLIT project as an exemple of good practices.
There are currently three versions of the database:
– the «old» Neo4J
– the «up to date» bdd_centrale.csv
– the work based on Pierre’s orthographic normalization
As an estimate, we will have an additional 150,000 observations to add to the existing 380,000.
6. Treatment of merchandises
We then have a lively discussion on how best to proceed for the treatment of goods.
Pierre suggests putting everything back to singular in orthographic normalization. We also discuss an automatic treatment of singulars and plurals automatic. automatise the calculus on the database to calculate singular and plural. This is put off till latter. More details follow :
– tout remettre au singulier: pour la lecture c’est peu attrayant, mais ça serait une solution si nous continuons à essayer de rentrer les marchandises dans la normalisation orthographique « au fil de l’eau ». Solution rejetée dans la discussion parce que a) reconvertir est trop lourd (Loïc) b) de tout façon faire « au fil de l’eau » est trop compliqué (moi)
– d’où la solution choisie: un petit algorithme à appliquer une fois la normalisation terminée, qui vérifie que ce qui est au singulier doit bien être au singulier, et ce qui est au pluriel doit bien être au pluriel. Sous le contrôle de Guillaume, voici ce que ça donnerait:
pour le mot x
if forme normalisée orthographique = forme singulier du dictionnaire
trouver forme plurielle dans dictionnaire
calculer N occurrences singulier et N occurrences pluriel standard dans base normalisée
calculer % N pluriel / (N pluriel + N singulier)
comparer résultat à choix; changer si le résultat ne correspond pas à la règle choisie (que je rappelle: pluriel seulement quand 75% de pluriel)
if forme normalisée orthographique = forme plurielle du dictionnaire
trouver forme singulier dans dictionnaire
calculer N occurrences singulier et N occurrences pluriel standard dans base normalisée
calculer % N pluriel / (N pluriel + N singulier)
comparer résultat à choix; changer si le résultat ne correspond pas à la règle choisie (idem, 75% pluriel => pluriel)
if forme normalisée orthographique ne correspond à aucune forme, arrêt et avertissement
La seule question est celle de la constitution d’un dictionnaire, mais ça ne me semble pas insoluble. Et effectivement c’est à faire un fois les nouvelles lignes rentrées.
Considering the delays (we need a functioning treatment of goods for the end of March), we discuss wether we should bypass «simplification» and link directly orthographic normalization with the 1789 classification (200/280 items). That latter task seems very ambitious. We decide to go through simplification (even though the resulting file (between 12,000 and 16,000 lines) still seems very long for the end user) and link with the sitc classification.
We discuss two additional points regarding simplification.
– Shall we keep the «qualities» ? We answer yes as we want not to loose any information at that stage. «harengs séchés» is not the same as « harengs».
– Shall we keep «categories» ? We answer no when they do not add any information. «poissons harengs» is the same as « harengs» There are two issues with simplification; every product is associated to qualities and to categories,
• is product plus quality the same as just product?
• should we keep categories of should we leave them and keep them only in classification?
We should try to think who the users of the data will be and work on data accordingly
7. Gregory’s work
As the meeting is reaching an end, Gregory is simple sent a number of pdfs to illustrate the kind of visualization we need.
The pre-prod version will be available soon. Please ask Guillaume for access.
Performance issues have been dealt with.
What should we add to the website to be a good tool for research?
In the simplification connection graph: get rid of the products with too few linkages (less than 4) to have those with more linkages clearer and more readable.
Another option is to filter by classification ?
Medicaments: change color of node to reflect hierarchisation ?
9. Chicago : no news. Things should coalesce when Loïc will be there, in March.
Meeting Toflit18 – 29/01/2016
Loïc Charles, Guillaume Daudin, Béatrice Dedinger, Federico D’Onofrio, Paul Girard, Jérôme Hericourt, Emmanuelle Lavallée, Ivan Ledezma, Guillaume Plique, Samuel Sanchez, Grégory Tible, Elisa Tirindelli
1 – Discussion around RICardo
2 – Back to Manylines
3 – New data exploration tools for Toflit (with,380 000 trade flows and 44,000 different goods and a number of different partners)
There are for the time being 4 ways to get into the data in the tool.
This is where one can treat the multiplicity of names in the data. For each case (goods and parters for the time being), there are at least three steps : orthographic normalisation // simplification (with no loss of information) // categorization (with some information loss)
How to update a classification ?
– Choose one (e.g. simplification)
– Export it in csv
– Open with LibreOffice (UTF-8)
– Modify the three-column file (last 2 columns)
– Import the csv
– This generates an «import report» about the links with the further classifications (e.g. changing «simplification» will have an effect on «categorization»
– Shall we do a versioning ?
– How to treat the «import report» ?
This is where one can see the available flows in the data
One can select the type of breakdown (in addition to year) : (direction, source, goods/partner classificatons, goods/partner simplication, grouping (product ou country)).
This yields 2 graphs :
Number of different groups per year
Number of flows per year
This is where one can see graphs with time series
On can make a selection according to :
– Source Type
– Kind (import, export, total)
Two curves are created :
– Number of flows by year
– Sum of value by year
There can be up to 6 choices.
For the time being, it is a bit slow, but results are put in a cache to be reused easily
This is were on can visualize graphs. The only one available is between directions and partners. The algorithms clusterizes geographical areas.
Latter in the meeting, we realize that it would be useful to be able to link goods and directions and not only partners and directions.
It would be nice also if a term network could be included.
4 – Paper presentations
4.1 Elisa on the analysis of bilateral trade flows between France and Hamburg between 1750-1789
Elisa Presentation_29 01 2016
French dataset (in 2014) : 3407 flows aggregate flows (Hamburg is often included in the North)
German dataset 1609 export flows between 1733-1798
Issues with both datasets : various units of measurement (even for the values) // linking products (changing named) // difficulty with prices and conversion rates
Solutions : work on common years / only on exports
The work that has been done : corrélation of the data in both sources. The correlation is rather strong (0.93 for the evolution of total flows, 0.98 for the breakdown by big sectors, 0.81 for the breakdown by products.
New exploration tools : linear regression and testing the Benford’s law
New questions on exogenous shocks : Seven year war, American revolution war, Interruption of commerce between France and Hamburg (1761-1769), trade liberalisation 1780.
To sum up : how reliable is the data ? What is the effect of wars ? What is the effect of treaties ?
There is a discussion on the share of Hamburg in the North and the importance of discrepancies in Hamburg.
4.2 Federico on Grain trade 1725-1821
Federico Presentation_29 01 2016
Only 6000 trade flows, broken down in 5 categories : Froment / other grains / third class grains / panifiable / transformed goods.
Directions : 6 directions with between 200 and 300 flows in each (only maritime)
Origins : Italie and the Baltic
Exports : colonies, Espagne, Italie, Nord…
Time series : : the food crisis are very apparent (70-71/89/1817)
Extending the national series: repeated sale method
Distribution of grain trade : Marseilles is the main export port by a large margin.
Links between the grain trade and population growth ? Role of riots ? The strongest correlation is between riots and t-1 exports. Is that because exports are in the summer and riots in the next spring ? Or because past exports stoke the fear for new permits ?
Projects : look at the geography of trade through GIS ? look at networks ? look at DEMETO models : demography and climate ? Thinking of co-occurences leads us to suggest that in the global view one should be able to link goods and directions and not only partners and directions.
4.3 Ivan and Emmanuelle on international production network in the 18th century Evidence from the French data (abstract)
The breakdown of the value added chain is not a 20th century innovation. They look at the sugar network. French per capita consumption is only 1/10th of English one.
The study of this international value-added chain is interesting. Especially the movement of the location of sugar transformation (which is a high VA activity).
One way to look at it is too look at the words associated with sugar.
The next meeting is at 9.30 of February 19th at the médialab.
Attending: Loïc Charles, , Guillaume Daudin, Béatrice Dedinger, Pierre Gervais, Paul Girard, Ivan Ledezma, Guillaume Plique,
1. Classifications in Neo4J
We now have a beta version of the algorithm that allows to change
«root» classifications while keeping as many links as possible with
We discuss how to best format the result of the algorithm so that it
can be used as a basis for further work
It will be tested on January 21st by Guillaume, Guillaume, Federico and Paul.
2. New Data
No news from Pierre, nor Lyon, nor Dunkerque.
No news from Marseille
The Eden treaty has been done
Gabriel and Torsten are progressing.
Cyril will treat Bordeaux.
Laura and Loïc’s interns will finish 1789.
Bordeaux, Rouen and Caen have been completed.
Tax information will not be included in Neo4J.
When have to think about a plan for the step by step integration of
new data to make the task manageable.
Nothing new. Things are on track.
On January 21st, Guillaume, Guillaume and Paul will test the algorithm (see point 1) On January 27th and 28th, the médialab will do a development meeting on data exploration. Gregory will be there. On January 29th (all day) there is a «big» TOFLIT meeting (9.30 at the Medialab). The next regular meeting will take place on February 19th, 9.30.
Thank you to Béatrice who has prepared these meeting minutes !
Atending: Loïc Charles, , Guillaume Daudin, Béatrice Dedinger, Pierre Gervais, Paul Girard, Ivan Ledezma, Guillaume Plique, Grégory Tible.
The Toflit sprint has led to the creation of three “views” by the developer team:
Updating the list of products (with the inclusion of Pierre’s data for example) has consequences on the whole chain from orthographic normalization to simplification and categorization.
Many cases: new items, renamed items, moved items, dropped items, new groups, splitted groups, suppressed groups (?).
– Paul and Guillaume are working on an algorithm to simplify the updating
– Pierre will finish orthographic normalization by December 19th.
– The algorithm will be written and tested before next meeting (January 8th)
No news of Lyon.
Paul will ask Pierre for the pictures in Bayonne’s archives.
Loïc will visit Dunkerque soon.
Loïc believes everything can be finished in 2016.
Work done: Bordeaux, 1789 and 1792, colonies.
Work to be done: Marseille (post-1765), 1787 Eden treaty.
We need to get Gabriel’s files
Torsten will do some work on Nantes.
Laura will work three months as an auto-entrepreneur to retranscribe herself and direct the work of two of Loïc’s interns (in June and July)
1789/1792 and tho colonies have been treated
Bordeaux is ready : Guillaume needs to check the integrate the data
TOFLIT: Guillaume Plique’s contract will end at the end of January. As we might not have to pay tho «prime de précarité», we will be able to hire Grégory Tible.for a 5-month internship.
RICardo : Grégory Tible work on RICardo is coming to its end.
A new intern will be hired in February to work on RIC entities and the data visualization.
The next monthly TOFLIT meeting will take place on January 8th 2016 at 9:30 am at Médialab. The next one will be on February 19th.
There will be a TOFLIT development workshop on January 27-28th
There will be a full TOFLIT meeting on January 29th. In the morning, the Médialab will present its work. In the afternoon, we will discuss research projects in the perspective of the Chicago meeting in June 2016. Researchers will present a one or two page presentation of a research project, and we will discuss, among other things, feasibility and data needs.
Attending : Cyril Canet, Loïc Charles, Guillaume Daudin, Béatrice Dedinger, Pierre Gervais, Guillaume Plique, Grégory Tible
The abstract of the paper on design choices has been submitted to the Digital Humanities 2016 conference.
The methodological paper has been submitted to « Historical Methods»
The database and the documentation will be finished before the end of Gregory’s internship.
Guillaume Plique présents a new interface for managing the classifications.
A 3-day sprint will take place on Novembre 16th to 18th to kick-start the work on the data exploration module.
4. Classification of goods
Pierre Gervais progresses in the orthographic normalization. In the end, he estimates that between 5 and 10 % of the terms will still not be identified. However, that will be a much smaller part of the database.
He plans to finish on november 20th (computer issues have since made that impossible).
We discuss the simplification.
– No progress from Thasectra, which should still do imports in Marseilles
– For the rest, the situation in the same as in September.
– Gabriel and Torsten will each do one year in Nantes.
– We need to find a way to retranscribe some more data
– Loïc is working on Bordeaux
– Guillaume Daudin will do Jérémy’s 1789 (National per direction) and 1792.
We still need to do Marseille, the colonies, the Eden treaty and the 1789 done by Anthony Rebours.
7. New data
We will ask Lyon again. Loïc is going to Dunkerque. We need someone for a picture in Bayonne : Paul has a suggestion (Pierre).
8. Cyril Canet
We discuss with Cyril Canet what he can do for the project during his stay in South Africa.
9. Werner Scheltjens
Werner will be invited in Dauphine in March 2016. At that point we will treat the issue of quantities.
10. Meeting schedule
TOFLIT Sprint : November 16th-18th
Monthly TOFLIT meetings : December 11th and January 8th (9.30 each time)
« Full » TOFLIT meeting : January 29th.
Chicago TOFLIT meeting : June 2016
Octobre 8th 2015 meeting
Attending: Loïc Charles, Guillaume Daudin, Pierre Gervais, Guillaume Plique, Donato Ricci, Gregory Tible
Import and export procedure for the data has been done, except for measurement units and outsiders (i.e. merchandizes which are not in the French data source)
We discuss how to treat these outsiders. They will have a direct link with sources, but not with flows.
We discuss notes in classifications. No formal structure. In all cases, only one note field per classification.
GP presents the first version of the classification browsing interface.
We discuss how to treat the glossary.
We decide on the next features.
A new version of Ricardo is presented (shortly before Béatrice does the same in Berlin), and some tweaks to the interface are suggested.
3. Classification of goods
Pierre Gervais has treated between 1500 and 2000 lignes in a fortnight. He just finished « étoffes».
He believes he could finish the treatment of the current (33,000 lines long) list November 15th.
This will not include any new good introduced since June 2015 (at least 3000 of them). If anyone wants to study a specific sector before then (e.g. grains), he should :
– Ask Pierre Gervais to treat it in the 33,000
– Do a «sister» classification treating the extra data just for that
– This work will never be integrated in « Pierre’s classification
4. Digital Humanities 2016
A paper will be presented at the Digital Humanitiest 2016 conference, despite the fact it will be submitted at this point.
– No progress from Thasectra, which should still do imports in Marseilles
– For the rest, the situation in the same as in September.
– Bayonne is done. The most difficult was doing Nantes and Rennes, because of the lack of totals.
– To be done : 1792, Marseille, Eden treaty data, 1789, Colones and Bordeaux.
We discuss the specific issue of Bordeaux. Taxes are mentioned. We decide to add columns for their measurement units, their the volume it applies to, the unit taxe and the total tax (4 ones)
7. New Data
No news from Lyon and Dunkerque
1 page is missing each in Rouen, Bayonne, and Nantes. Maybe we should send an intern to take the pictures and do a bit of retranscription.
The next meeting is November 6th at 0930 at the Médialab.