FAQ: SuperMemo data exchange via XML

You can transfer your learning material between SuperMemos via XML
I have seen the phrase "XML file" all over the place. I did a Google search on it and got too many responses to even bother trying to read. Can you give me a quick and dirty?
When googling for general terms, always add "supermemo" to your search string. In the context of SuperMemo 2004 and later, "XML file" means: your learning material exported from SuperMemo with File : Export : XML (or other options that export in the XML format). Material in the form of the XML file, can be used, for example, to transfer your learning process from SuperMemo 2004 or later to SuperMemo for Pocket PC back and forth

Merging learning material on Pocket PC
(Bill Sanchez, Thu, Jun 20, 2002 4:22)
I am using SuperMemo CE on my Pocket PC. I have created questions in different files. I would like to combine all the questions into one file now
You could export your files as XML and merge the XML files. If you import it back to SuperMemo for Pocket PC, your collections will have been merged

Special repetition case in XML import
(Beta-piwo, Jul 05, 2004)
I got a "special repetition case" message in XML import. What does it mean? 
Interval Boost=1.3% 
Repetition Sequence=100% 
Interval protectively changed from: 21 to 20 
INTEGRATING {2}:Item #232566: The r�ntgen is a unit of dosage of X-rays or gamma radiation 
Reps: 1->2 
Lapses: 0->0 
Interval: 23->20 
Last rep: 12.06.2004->22.06.2004 
Next rep: 05.07.2004->12.07.2004 
UF: 23->2 AF: 3->3.274
This message will appear if you duplicate repetitions in two collections. It may also appear if you delay repetitions and import of your XML learning process over a longer period (beyond one month). In such cases SuperMemo will try to guess what has caused data discrepancy, take the optimum course of action so that to minimize the cost in learning time. 

In this specific case, SuperMemo compared the recency of the two repetitions, the change to item's interval, as well as the sequence of repetitions executed. In the end, it concluded (preponderance 62.19%), that the repetition update should be executed. However, it decided to reduce the interval from 23 days (in the source collection) and 21 days suggested by the XML file to 20 days. In the end, you will find the repetition recorded as executed on Jun 22, 2004 (as opposed to Jun 12) with the interval of 20 days, and the next repetition date scheduled on Jul 12, 2004

You can transfer between SuperMemo for Pocket PC and SuperMemo for Windows
(Miloslav Sk�cel, Czech Republic, Thursday, November 07, 2002 11:10 PM)
Is it possible to transfer my knowledge between SuperMemo for Pocket PC and SuperMemo for Windows?
Yes. XML data exchange makes it possible to exchange questions-and-answers as well as the learning process. See: Revolution in the Pocket for a detailed algorithm

XML data exchange will help transfer the learning process between platforms
(leopold16dk, Jun 13, 2002)
Is it possible to convert my learning process from SuperMemo for Palm Pilot to SuperMemo for Pocket PC?
Currently you can only transfer question&answer material between handheld platforms. However, a comprehensive XML-based data exchange for all SuperMemo desktop, handheld and on-line platforms is in design

Text filters in XML export
(Michael, Jun 14, 2004, 08:23:13)
What is the purpose of the filter dialog box in XML export?
Text filters in SuperMemo can be used to pre-process texts before using export options. Using filters is optional. Without filtering, the exported text will look exactly the same as the text in the collection

Error: Cannot guess the correct repetition history
Everyday I get 2-3 of these errors when importing my XML file from SuperMemo CE to SuperMemo 2008 (with File : Import : Learning process : XML file): 

Cannot guess the correct repetition history 
CANNOT INTEGRATE: Item #11553: Hier ist [a good] [...] Wein. 
Reps: 5->7 
Lapses: 0->0 
Interval: 22->60 
Last rep: 01.09.2005->23.09.2005 
Next rep: 23.09.2005->22.11.2005 
UF: 1.048->2.766 
AF: 5.383->5.378

SuperMemo 2004 or later allows of simultaneous repetitions of the same material in SuperMemo 2004 or later and on a Pocket PC device. It also attempts to heuristically fill gaps in repetition history if more than one repetition was made on Pocket PC. If reconciling data from two collections is not possible, it may display "Cannot integrate" error (in this case "Cannot guess the correct repetition history"). For example, it could happen if the interval between repetitions was negative or zero (e.g. if you use Midnight Shift in SuperMemo 2004 or later, etc.). 

Each time you get this error, you can either ignore it (repetition made on Pocket PC will be discarded) or go to the element in question and manually force a repetition (Ctrl+Shift+R) or choose a new interval (Ctrl+J). This would be a reason for worry only if a large proportion of items showed such problems (e.g. due to a wrong setting of the date in your computer).

Simultaneous repetitions are not recommended as these introduce some confusion in the learning process. You can move between platforms freely, but then you should either make repetitions on one platform only or use subsets to exclude Pocket PC repetitions from SuperMemo 2004 or later repetition set while working on Pocket PC

Problems with importing XML from Excel
(Stefan, Singapore, Jun 22, 2005, 14:42:29)
I want to add new words to Pocket PC by working on a desktop computer. I have opened an XML file from Pocket PC in Excel and added new words. After this I have transferred the XML file back to the Pocket PC, but the import did not work.
If you want to create your word pairs on a desktop computer, use Q&A file format. XML format is useful only if you want to create your material in SuperMemo 2004 or later. You cannot import XML files from Excel as it uses a completely different file definition (DTD). XML format definition is SuperMemo-specific and is used only for transferring files between different versions of SuperMemo

Generating SuperMemo XML files manually
(Xavier Verges Farrero, Apr 26, 2005, 00:40:32)
I would like to generate XML files myself. Is the ID only important within a given xml file, or is it related to some other data? Could I feed SuperMemo with generated xml files that would not have the LearningData elements?
You can produce XML files for SuperMemo using any tool of your choice. Element IDs are currently important only if you wish to re-import material exported from SuperMemo 2004 or later back to your SuperMemo 2004 or later collection. If you skip them, new elements will be created. If you skip LearningData, elements will be treated as pending (i.e. not learned)

Interval protectively changed (#11366)
(colin thorne, Nov 22, 2006, 15:00:18)
When I exchange data via XML with Pocket PC, does the phrase "Interval protectively changed from: 17 to 8" mean that the interval for the next repetition was formerly set to 17, but got reduced down to 8?
Yes. Because of differences in the algorithms used for spacing repetitions in SuperMemo for Windows and in SuperMemo for Pocket PC, minor difficulties in planning future repetitions arise. This is most visible when you make many repetitions on Pocket PC before re-importing to a Windows collection. As a result, when SuperMemo for Windows is not sure about the exact repetition history and the optimum next interval, it errs on the safe side and uses a shorter interval before the next repetition. All such changes to intervals are reported to you.

XML data exchange has been designed for smaller data sets (#1718)
(Wojciech Baranski, Poland, Monday, April 11, 2005 11:14 PM)
XML Export of my 6000 items Spanish database from SuperMemo 2004 takes 9 hours! The same process in SuperMemo for Pocket PC runs for 5-10 minutes. It seems to me, that the export algorithm has a bug. I'm starting to add new entries and I fear that exporting a database with 7500 entries will take more than 12 hours
SuperMemo 2004 slows down when building the XML tree because it uses Microsoft's native validating parser that builds the entire tree in memory before writing it to a file. The bigger the tree the longer it takes to verify and expand it. SuperMemo CE uses its own linear procedure for exporting XML data that is not size-dependent. XML data exchange was primarily designed for a quick exchange of small portions of your learning collection between platforms. As long as you export small subsets that span outstanding material for up to a week (usually no more than 1000-2000 items), the export shall take minutes (unless you are using PC less rich in resources). It is also not recommended to export large files from SuperMemo for Pocket PC as their size is again limited by available resources on Pocket PC. In your case, that would mean that you keep your large collection in Windows and learn its small portions on Pocket PC. You can also transfer the entire collection to Pocket PC once on the assumption you will not return to SuperMemo 2004