RESULTS OF THE SUPER-MEMO USER QUESTIONNAIRE (SEPTEMBER 1993)

The questionnaire, together with a standard newsletter and special price offer, has been sent to 800 users of SuperMemo that registered at SuperMemo World by the end of July 1993.

The following response rates have been achieved:
- returned, fully-completed questionnaires 13.0%
- Database Competition registration forms 5.0%
- orders for upgrades, databases and sound boards 7.4%
* orders for upgrades 6.7%
# from SuperMemo 6 to SuperMemo 7 4.1%
# from SuperMemo 6 to SuperMemo 6.7 1.6%
# from SuperMemo 7.0 to SuperMemo 7.1 0.5%
# from SuperMemo 5 to SuperMemo 6 0.5%
* orders for databases
# English Grammar 1.8%
# English Vocabulary by M.Cajza 1.6%
# First Certificate in English 1.4%
# Polish Orthography 1.3%
# Technical Mix 0.9%
# English Mix 0.8%
# Medical Mix 0.6%
# German 0.6%
# Biological Mix 0.5%
# Language Mix 0.1%
* orders for sound boards 0.3%
Total orders PLZ 27,000,000 (132 orders altogether)
68% revenue came from upgrades
32% revenue came from databases
Cost of sending about PLZ 3,000,000.
Return on investment 900%

The average user has been learning with SuperMemo for 10 months, spending 27 minutes on repetitions per day. He or she planned to increase the learning time in the future by 10 minutes to 37 minutes per day.
The average learner had 4400 items memorized in his or her database.
The average knowledge acquisition rate was 391 items/year/minute, or 263 items/year/minute after rejecting scores above 600 and below 100 items/year/minute.
For a learner who spends 30 minutes per day on repetitions, this learning speed allows to master Basic English (3000 items) in 93 days, and Advanced English (38,000 items) in 3 years and 3 months. One should remember, however, that the assumption of the constant value of the acquisition rate is only a rough approximation. As the data have been collected for the 10 month period (on average), the Basic English could be memorized much faster, while it would take a little bit longer to memorize Advanced English.

Breaks in learning have not been considered in the above calculations, though it is know that they took about 17% of the learning time on average. Obviously, without breaks, learning might proceed much faster.

The following are acquisition rates for different age groups and for different educational backgrounds:

University education 343 items/year/minute
Secondary education 114 items/year/minute
Secondary and university students 549 items/year/minute

Age below 25 494 items/year/minute
Age from 25 to 35 373 items/year/minute
Age over 35 272 items/year/minute

Users with different versions of SuperMemo:
SuperMemo 2 1%
SuperMemo 5 10%
SuperMemo 6 78%
SuperMemo 7 11%

The learners range from 16 to 61 in age. The average age was 33.
29% of the learners were above 40 years of age, and 15% of learners
were below 20.

In the grade point scale from 0 to 5 (5-excellent, 4-good, 3-pass, etc.),
the learners graded the learning method used in SuperMemo at 4.75, and
SuperMemo software at 4.17. Grades have not been significantly correlated
with the age, but they were slightly dependent on the learners education:

Method Software
MSc or more 4.67 4.09
university students 4.88 4.31
secondary education 4.78 4.22
high school students 4.64 3.78

The following were the opinions about the method:
excellent aid in learning 74.5%
useful idea 25.5%
nothing special 0.0%
artificial approach to learning 0.0%
waste of time 0.0%
The following were the opinions about the program:
excellent product 32.7%
reliable product 52.5%
useful product that must yet be
improved 14.8% (only users of SM6 or less)
useless program 0.0%

The highest ranked databases were (standard database scale from 0 - worst
to 10 - best) Advanced English 7.95 and Basic English 7.45.

The following was the educational background of users of SuperMemo:
university education 62%
university students 16%
secondary education 9%
high school students 14%
primary education 0%
primary school students 0%
Please note, however, that many of the users indicated that their children,
usually at the primary level, often used SuperMemo as well.

The following categories were listed as the profession of the learner:
engineer 18%
student 13% (among these: computer science 19%,
medicine 18%, electronics 7%)
high school student 11%
computer scientist 10%
physician 8%
acad. teacher/scientist 6%
private entrepreneur 5%
technician 5%
teacher 4%
economist 4%
civil servant 4%
agronomist/farmer 4%
policeman/fire fighter 1%
soldier less than 1%
unemployed
priest

The user indicated the following as the source of his first contact with
SuperMemo:
Press 79%
Shareware, illegal copy, friend 10%
Shop 7%
Fairs 3%
TV 0%

The press was the main source of knowledge about SuperMemo (84% of this was computer press):
PC Kurier 37%
Enter 19%
Computer World 7%
Bajtek 3%
Gazeta Wyborcza 2%
PC World Komputer 1%
Informatyka 1%

6% of users admitted having used an illegal copy before buying SuperMemo.

The following was mention as the most important strategic direction for development of SuperMemo:

  1. developing more databases 56% (SuperMemo Library, 1998)
  2. developing more audio databases 42%
  3. developing national versions of software 40%
  4. programmable SuperMemo 34%
  5. developing more graphic databases 31%
  6. compression of sound and graphic files 22%
  7. adding hypertext capability 20% (SuperMemo 8, 1997)
  8. developing new SuperMemo Typing Tutor 19%
  9. adding multi-font support 18% (SuperMemo 98)
  10. OLE 2.0 13% (SuperMemo 98)
  11. SuperMemo for Windows NT 9% (SuperMemo 98)
  12. Adding video capability 9% (SuperMemo 8, 1997)
  13. Adding virtual reality capability 7%


Answers to the question: "What do you like most in SuperMemo?":
its effectiveness 63%
the fact that it enforces systematic work 15%
possibility to create one's own databases 13%
ease of use 12%
user interface 5%
statistical options 4%
fast development of SuperMemo software 3%
minimum information principle 3%
trustworthiness of SuperMemo World 3%
work with SuperMemo is not boring 2%
self-assessment principle 2%
no need to type in the answer 1%
other likes: ability to edit fonts, ability to define one's own
keyboard (SM6), improving memory, forgetting curves, originality,
pronunciation in databases, manual, minimum number of options,
Basic English, reliability of software, reasonable pricing, and
fast growth of SuperMemo World, and more
NO OPINION 3%

Answers to the question: "What do you dislike most in SuperMemo?":
no dislikes at all! 20%
incomprehensible Process/Analysis menu 11%
no possibility to type in answers 8%
lack of Polish version 4%
too few databases 4%
no confirmation at registering SuperMemo 4%
long response time to orders at SMW 4%
newsletters should be more frequent 3%
fixed size of windows 2%
unavailability of SuperMemo in shops 2%
tools are not integrated with the program 2%
self-assessment principles 2%
Dislikes which refer only to SuperMemo 6:
user interface (colors, mouse, etc.) 13%
bad translation of the manual 5%
problems with Polish fonts 4%
difficult to use 3%
no audiovisual databases 2%
poor item editor 2%
poor database safety 1%
no toolkit 1%
Other dislikes: Lapses cannot be less than 3%, databases are not upgradeable
(e.g. when one gets new Advanced English), featuritis (too many options),
items are limited in size, manual does not describe the newly introduced
features, accepting items with Esc, no IPA pronunciation in databases,
price, no programmable vacation break, no database development toolkit,
no fractional grades, poor quality of databases, the need to work
systematically, no tools to import TXT or DBF files, and more
NO OPINION 4%

The proposed improvements to SuperMemo software are listed below. In parentheses, the name of the person who was first to propose or described the new feature in most detail. Also, after the exclamation point, remarks on the existing features:

  1. 8% possibility to type in the answer (Spell-Pad in SuperMemo 7.4)
  2. 4% clearer manual (e.g. info on how to learn, Analysis, etc.)
  3. 4% allow sizing Q&A fields (SuperMemo 7.3)
  4. 4% allow importing TXT and DBF (also Lotus, Framework, etc.)
  5. 4% allow turning off some statistical options (Zbigniew Dziubinski)
  6. 3% integrate Toolkit (SuperMemo 7.4)
  7. 3% more pronunciation in databases (Advanced English 97)
  8. 3% selectable key for accepting items (not Esc) (Teresa Kustusz)
  9. 3% add mouse support to SuperMemo 6
  10. 3% block Alt-M in SM6 (possible in SuperMemo 6.7)
  11. 2% explain Analysis (give examples)
  12. 2% configuration file for different users using the same SuperMemo in a network (Leszek Danczak)
  13. 2% plan a break in learning
  14. 2% correct bugs in databases
  15. 2% password on opening a database (Wojciech Kopczuk)(SuperMemo 7.5)
  16. 2% transferring items between databases (Piotr Bajerski) (SuperMemo 7.3)
  17. 2% add macro-capability (Aleksander Spalek)
  18. 1% selectable first interval in Memorize (Tadeusz Fryska) (SuperMemo 7.3)
  19. 1% print subset of items (Rafal Duszkiewicz)
  20. 1% merge databases (possible with TRANSFER.EXE)
  21. 1% data interpretation "for people" e.g. verbal description
  22. 1% click E-distribution and view the items (Piotr Kowal)
  23. 1% change the order of items in a database (Rafal Duszkiewicz) (SuperMemo 7.3)
  24. 1% optional end-of-learning report
  25. 1% Day counts the process, not the age of the database (Adam Jasinski)
  26. 1% database name as a parameter (Adam Pawlowski) (SuperMemo 7.1)
  27. 1% tool for converting databases to dictionaries (Rafal Duszkiewicz)
  28. 1% sound in SM6, e.g. Sound Galaxy
  29. 1% repetition priority for databases (Piotr Bajerski)
  30. 1% random test on a subset, e.g. memorized items (Bogdan Kowalewski) (SuperMemo 7.3)
  31. 1% print all outstanding repetitions
  32. 1% mark databases and let only a friend use them (Roman Roszak)
  33. 1% do not abandon DOS (we consider writing SuperMemo 7 for DOS and wait for opinions)
  34. 1% different keyboard setting for Q&A
  35. 1% allow swapping questions with answers (possible globally with SWAP.EXE, also implemented locally in 7.3 as Edit:Swap)
  36. 1% add a stop-watch to the program; limit Learn and Expand times (Przemyslaw Pempera)(SuperMemo 99)
  37. 1% translate to Polish (implemented in 6.5PL and 7.2PL)
  38. 1% relational database operations (Marek Ziobro)
  39. 1% print a database report (Rafal Duszkiewicz) (possible with CROSS.EXE)
  40. 1% limit the no of repetitions per day (Tadeusz Fryska)
  41. 1% histogram with the number of lapses (Adam Pawlowski)
  42. 1% count the total working time with the database
  43. 1% tool for eliminating identical items
  44. 1% quit-the-program report (Aleksander Spalek)
  45. 1% allow running in Windows 3.0 (because of various limitations of Windows 3.0, this proposition will not be considered)

Other propositions:
- several people use the same ITM file (Tadeusz Fryska)
- repetition priority for items (Piotr Bajerski)
- global font and keyboard setup for all databases (Tadeusz Fryska)
- view items by clicking the bar in all histograms (Adam Pawlowski)
- better database security (implemented in 7.3)
- develop SuperMemo hand-held (Window CE version; September 1999)

Proposed promotional activities:
26% promote at schools (primary and secondary)
11% more newsletters to users
10% shorten customer response time (e.g. waiting 2 months for a package)
10% promote at universities
10% more adverts in general press
9% lower the price
8% more databases
8% more aggressive advertising
7% more dealers (program is not available)
7% large discounts for educational institutions
6% promote in language schools
6% more emphasis on shareware
6% more adverts in technical press
5% more adverts in the press for teachers
5% advertise examples of people who succeeded thanks to SuperMemo
4% SUPER-MEMO IS PROMOTED WELL ENOUGH
4% more multimedia databases (animals, history, geography, singing birds)
4% large discounts for students
4% constantly develop software
3% send info immediately upon receiving the registration card
3% good discounts for language schools acting as dealers
3% distribute shareware databases (e.g. via BBSes, or shareware mixes)
3% demo in every shop
3% advertise in local press
2% simpler SuperMemo for younger users
2% set up a FIDO conference
2% release SuperMemo in Polish
2% recommendation sale
2% posters in every shop
2% organize competitions (e.g. Best Database Competition)
2% OEM deals
2% more emphasis on the effectiveness of the method
2% more adverts in the press for physicians
2% make demonstration meetings with potential customers
2% educate dealers
2% do not abandon DOS version
2% direct mail to schools
2% direct mail to new buyers of computers
2% direct mail to all institutions with PCs
2% concentrate on Windows
2% cheaper upgrades
2% better quality of databases (employ experts)
2% better customer service
2% advertise on TV
2% add demo disks to journal editions (e.g. Enter)
1% translate databases to many languages
1% SuperMemo free for schools
1% send free copies to press for review
1% reduce prices for older versions
1% promote at courses for users of PCs
1% posters in all educational institutions
1% personal contact with schools (e.g. of distributors)
1% paid promotion by the users
1% more understandable leaflet
1% more realistic statistics of the speed of learning
1% more presence at fairs
1% more info about the program itself in leaflets
1% more free databases in the package
1% more databases for secondary schools
1% leaflets in every shop
1% involve schools in massive development of databases
1% emphasize that commercial databases are not needed to use the program
1% emphasize cheaper upgrades and databases for legal users
1% demo of the newest releases for the users
1% database samples for registered users
1% courses on SuperMemo
1% contact authors of books about computers, add shareware disk
Other propositions: advertise in Wprost, advertise in Gazeta Wyborcza, address also those who have no PCs (let them buy a PC for SuperMemo), arrange a program about SuperMemo in Eureka on TV, etc.