Downloaded 277 times
![MongoDBis schema-lessJSON-style datastoreEachdocumentcanhaveitsownschemaDocumentsinside a collectionusuallyshare a commonschemabyconvention{‚name‘ : ‚kate‘, ‚age‘:12, }{‚name‘ : ‚adam‘, ‚height‘ : 180}{‚q‘: 1234, ‚x‘ = [‚foo‘, ‚bar‘]}](https://image.slidesharecdn.com/python-mongodb-training-europython-2011-110620230438-phpapp02/75/Python-mongo-db-training-europython-2011-12-2048.jpg)
![Arrays (1/2)Like (nested) lists in Pythondb.foo.insert({colors: [‚green‘, ‚blue‘, ‚red‘]})db.foo.find({colors: ‚red‘})Useindexes](https://image.slidesharecdn.com/python-mongodb-training-europython-2011-110620230438-phpapp02/75/Python-mongo-db-training-europython-2011-52-2048.jpg)
![Arrays (2/2) – matchingarraysdb.bar.insert({users: [ {name: ‚Hans‘, age:42}, {name:‘Jim‘, age: 30 }, ]})db.bar.find({users : {‚$elemMatch‘: {age : {$gt:42}}}})](https://image.slidesharecdn.com/python-mongodb-training-europython-2011-110620230438-phpapp02/75/Python-mongo-db-training-europython-2011-53-2048.jpg)
![Mongokit (2/3)classBlogPost(Document):structure = { 'title': unicode, 'body': unicode, 'author': pymongo.objectid.ObjectId, 'created_at': datetime.datetime, 'tags': [unicode], }required_fields = ['title','author', 'date_creation']blog_post = BlogPost()blog_post['title'] = 'myblogpost'blog_post['created_at'] = datetime.datetime.utcnow()blog_post.save()](https://image.slidesharecdn.com/python-mongodb-training-europython-2011-110620230438-phpapp02/75/Python-mongo-db-training-europython-2011-58-2048.jpg)
![MongoAlchemy(2/2)frommongoalchemy.documentimportDocument, DocumentFieldfrommongoalchemy.fieldsimport *fromdatetimeimportdatetimefrompprintimportpprintclass Event(Document):name = StringField()children = ListField(DocumentField('Event'))begin = DateTimeField() end = DateTimeField()def __init__(self, name, parent=None):Document.__init__(self, name=name)self.children = []ifparent != None:parent.children.append(self)](https://image.slidesharecdn.com/python-mongodb-training-europython-2011-110620230438-phpapp02/75/Python-mongo-db-training-europython-2011-63-2048.jpg)
Uploaded byAndreas Jung
Python mongo db-training-europython-2011
This document provides an overview of using Python and MongoDB together. It discusses MongoDB concepts and architecture, how to get started with MongoDB using the interactive console, and basic CRUD operations. It then covers installing and using PyMongo, the main Python driver for MongoDB, and some popular high-level Python frameworks built on top of PyMongo like MongoEngine and MongoAlchemy.
More Related Content
![[2D1]Elasticsearch 성능 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/2d1elasticsearch-140929192211-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Python mongo db-training-europython-2011
Python mongo db-training-europython-2011
- 1. Python andMongoDBThe perfectMatchAndreas Jung, www.zopyx.com
- 2. Trainer Andreas JungPythondevelopersince 1993Python, Zope & PlonedevelopmentSpecialized in Electronic PublishingDirectoroftheZopeFoundationAuthorofdozensadd-onsfor Python, ZopeandPloneCo-Founderofthe German Zope User Group (DZUG)Member ofthePloneFoundationusingMongoDBsince 2009
- 3. Agenda (45 minutesper slot)IntroductiontoMongoDBUsingMongoDBUsingMongoDBfrom Python withPyMongo(PyMongoextensions/ORM-ishlayersor Q/A)
- 4. Things not coveredinthistutorialGeospatialindexingMap-reduceDetails on scaling (Sharding, Replicasets)
- 5.
- 6.
- 7. Let‘sagree on thefollowingorleave...MongoDBiscoolMongoDBis not the multi-purpose-one-size-fits-all databaseMongoDBisanotheradditionaltoolforthesoftwaredeveloperMongoDBis not a replacementfor RDBMS in generalUsetherighttoolforeachtask
- 8. And.....Don‘taskmeabouthowto do JOINsin MongoDB
- 9. Oh, SQL –let‘shavesomefunfirstA SQL statementwalksinto a bar andseestwotables. He walksandsays: „Hello, may I joinyou“A SQL injectionwalksinto a bar andstartstoquotesomething but suddenlystops, drops a tableanddashes out.
- 10. The historyofMongoDB10gen foundedin 2007Startedascloud-alternative GAEApp-engineedDatabase pJavascriptasimplementationlanguage2008: focusing on thedatabasepart: MongoDB2009: firstMongoDBrelease2011: MongoDB 1.8:Major deploymentsA fast growingcommunityFast adoptationfor large projects10gen growing
- 11.
- 12. MongoDBis schema-lessJSON-style datastoreEachdocumentcanhaveitsownschemaDocumentsinsidea collectionusuallyshare a commonschemabyconvention{‚name‘ : ‚kate‘, ‚age‘:12, }{‚name‘ : ‚adam‘, ‚height‘ : 180}{‚q‘: 1234, ‚x‘ = [‚foo‘, ‚bar‘]}
- 13. Terminology: RDBMS vs.MongoDB
- 14. CharacteristicsofMongoDB (I)High-performanceRich querylanguage(similarto SQL)Map-Reduce (ifyoureallyneedit)SecondaryindexesGeospatialindexingReplicationAuto-sharing (partitioningofdata)Manyplatforms, driversformanylanguages
- 15. CharacteristicsofMongoDB (II)Notransactionsupport, onlyatomicoperationsDefault:„fire-and-forget“ modefor high throughput„Safe-Mode“: waitforserverconfirmation, checkingforerrors
- 16. TypicalperformancecharacteristicsDecentcommoditiyhardware:Upto 100.000 read/writesper second (fire-and-forget)Upto 50.000 reads/writes per second (safemode)Yourmileagemayvary– depending onRAMSpeed IO systemCPUClient-sidedriver& application
- 17.
- 18. MongoDB: Pros &Cons
- 19. DurabilityDefault: fire-and-forget (usesafe-mode)Changesarekeptin RAM (!)Fsynctodiskevery 60 seconds (default)Deploymentoptions:Standaloneinstallation: usejournaling (V 1.8+)Replicated: usereplicasets(s)
- 20. Differences from TypicalRDBMSMemory mapped dataAll data in memory (if it fits), synced to disk periodicallyNo joinsReads have greater data localityNo joins between serversNo transactionsImproves performance of various operationsNo transactions between servers
- 21. Replica SetsCluster ofN serversOnly one node is ‘primary’ at a timeThis is equivalent to masterThe node where writes goPrimary is elected by concensusAutomatic failoverAutomatic recovery of failed nodes
- 22. Replica Sets -WritesA write is only ‘committed’ once it has been replicated to a majority of nodes in the setBefore this happens, reads to the set may or may not see the writeOn failover, data which is not ‘committed’ may be dropped (but not necessarily)If dropped, it will be rolled back from all servers which wrote itFor improved durability, use getLastError/wOther criteria – block writes when nodes go down or slaves get too far behindOr, to reduce latency, reduce getLastError/w
- 23. Replica Sets -NodesNodes monitor each other’s heartbeatsIf primary can’t see a majority of nodes, it relinquishes primary statusIf a majority of nodes notice there is no primary, they elect a primary using criteriaNode priorityNode data’s freshness
- 24. Replica Sets -NodesMember 1Member 2Member 3
- 25. Replica Sets -Nodes{a:1}Member 1SECONDARY{a:1}{b:2}Member 2SECONDARY{a:1}{b:2}{c:3}Member 3PRIMARY
- 26. Replica Sets -Nodes{a:1}Member 1SECONDARY{a:1}{b:2}Member 2PRIMARY{a:1}{b:2}{c:3}Member 3DOWN
- 27. Replica Sets -Nodes{a:1}{b:2}Member 1SECONDARY{a:1}{b:2}Member 2PRIMARY{a:1}{b:2}{c:3}Member 3RECOVERING
- 28. Replica Sets -Nodes{a:1}{b:2}Member 1SECONDARY{a:1}{b:2}Member 2PRIMARY{a:1}{b:2}Member 3SECONDARY
- 29. Replica Sets –Node TypesStandard – can be primary or secondaryPassive – will be secondary but never primaryArbiter – will vote on primary, but won’t replicate data
- 30. SlaveOkdb.getMongo().setSlaveOk();Syntax varies bydriverWrites to master, reads to slaveSlave will be picked arbitrarily
- 31.
- 32. ShardA replica setManagesa well defined range of shard keys
- 33. ShardDistribute data acrossmachinesReduce data per machineBetter able to fit in RAMDistribute write load across shardsDistribute read load across shards, and across nodes within shards
- 34. Shard Key{ user_id:1 }{ lastname: 1, firstname: 1 }{ tag: 1, timestamp: -1 }{ _id: 1 }This is the default
- 35. MongosRoutes data to/fromshardsdb.users.find( { user_id: 5000 } )db.users.find( { user_id: { $gt: 4000, $lt: 6000 } } )db.users.find( { hometown: ‘Seattle’ } )db.users.find( { hometown: ‘Seattle’ } ).sort( { user_id: 1 } )
- 36. Differences from TypicalRDBMSMemory mapped dataAll data in memory (if it fits), synced to disk periodicallyNo joinsReads have greater data localityNo joins between serversNo transactionsImproves performance of various operationsNo transactions between serversA weak authentication and authorization model
- 37. Part 2/4UsingMongoDBStartingMongoDBUsingtheinteractive MongoconsoleBasic databaseoperations
- 38. Gettingstarted...theserverwget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-1.8.1.tgztarxfzmongodb-osx-x86_64-1.8.1.tgzcd mongodb-osx-x86_64-1.8.1mkdir/tmp/dbbin/mongod –dbpath /tmp/dbPick upyour OS-specificpackagefromhttp://www.mongodb.org/downloadsTake careof 32 bitbs. 64 bitversion
- 39. Gettingstarted...theconsolebin/mongodmongodlistenstoport 27017 bydefaultHTTPinterface on port 28017> help> db.help()> db.some_collection.help()
- 40. Datatypes...Remember: MongoDBis schema-lessMongoDBsupportsJSON + some extra types
- 41.
- 42. Inserting> db.foo.insert(document)> db.foo.insert({‚firstname‘: ‚Ben‘})everydocumenthas an „_id“ field„_id“ insertedautomaticallyif not present
- 43. Querying> db.foo.find(query_expression)> db.foo.find({‚firstname‘: ‚Ben‘})Queriesareexpressedusing JSON notationwith JSON/BSON objectsqueryexpressionscombinedusing AND (bydefault)http://www.mongodb.org/display/DOCS/Querying
- 44. Queryingwithsorting> db.foo.find({}).sort({‚firstname‘ :1,‚age‘: -1})sortingspecification in JSON notation1 = ascending, -1 = descending
- 45.
- 46. Updating> db.foo.update(criteria, obj,multi, upsert)update() updatesonlyonedocumentbydefault (specifymulti=1)upsert=1: ifdocumentdoes not exist, insertit
- 47.
- 48. Updating> db.foo.update(criteria, obj,multi, upsert)update() updatesonlyonedocumentbydefault (specifymulti=1)upsert=1: ifdocumentdoes not exist, insertit
- 49. Removingdb.foo.remove({}) // remove alldb.foo.remove({‚firstname‘ : ‚Ben‘}) // removebykeydb.foo.remove({‚_id‘ : ObjectId(...)}) // removeby _idAtomicremoval(locksthedatabase)db.foo.remove( { age: 42, $atomic : true } )http://www.mongodb.org/display/DOCS/Removing
- 50. Indexesworkingsimilartoindex in relationaldatabasesdb.foo.ensureIndex({age: 1}, {background: true})onequery– oneindexCompoundIndexesdb.foo.ensureIndex({age: 1, firstname:-1}Orderingofqueryparametersmattershttp://www.mongodb.org/display/DOCS/Indexes
- 51. Embedded documentsMongoDBdocs =JSON/BSON-likeEmbeededdocumentssimilarnesteddicts in Pythondb.foo.insert({firstname:‘Ben‘, data:{a:1, b:2, c:3})db.foo.find({‚data.a‘:1})DottednotationforreachingintoembeddedocumentsUsequotesarounddottednamesIndexes work on embeddesdocuments
- 52. Arrays (1/2)Like (nested)lists in Pythondb.foo.insert({colors: [‚green‘, ‚blue‘, ‚red‘]})db.foo.find({colors: ‚red‘})Useindexes
- 53. Arrays (2/2) –matchingarraysdb.bar.insert({users: [ {name: ‚Hans‘, age:42}, {name:‘Jim‘, age: 30 }, ]})db.bar.find({users : {‚$elemMatch‘: {age : {$gt:42}}}})
- 54. Part 3/4UsingMongoDBfrom PythonPyMongoInstallingPyMongoUsingPyMongo
- 55. InstallingandtestingPyMongoInstallpymongovirtualenv –no-site-packagespymongobin/easy_installpymongoStart MongoDBmkdir/tmp/dbmongod –dbpath /tmp/dbStart Pythonbin/python> importpymongo> conn = pymongo.Connection(‚localhost‘, 27127)
- 56. Part 4/4? High-levelPyMongoframeworksMongokitMongoengineMongoAlchemy? Migration SQL toMongoDB? Q/A? Lookingat a real worldprojectdonewithPyramidandMongoDB?? Let‘stalkabout..
- 57. Mongokit (1/3)schemavalidation (wichusesimple pythontype forthedeclaration)dotednotationnestedandcomplexschemadeclarationuntypedfieldsupportrequiredfieldsvalidationdefaultvaluescustomvalidatorscrossdatabasedocumentreferencerandomquerysupport (whichreturns a randomdocumentfromthedatabase)inheritanceandpolymorphismesupportversionizeddocumentsupport (in betastage)partial authsupport (itbrings a simple User model)operatorforvalidation (currently : OR, NOT and IS)simple web frameworkintegrationimport/exporttojsoni18n supportGridFSsupportdocumentmigrationsupport
- 58. Mongokit (2/3)classBlogPost(Document):structure ={ 'title': unicode, 'body': unicode, 'author': pymongo.objectid.ObjectId, 'created_at': datetime.datetime, 'tags': [unicode], }required_fields = ['title','author', 'date_creation']blog_post = BlogPost()blog_post['title'] = 'myblogpost'blog_post['created_at'] = datetime.datetime.utcnow()blog_post.save()
- 59. Mongokit (3/3)Speed andperformanceimpactMongokitisalwaysbehindthemostcurrentpymongoversionsone-mandevelopershowhttp://namlook.github.com/mongokit/
- 60. Mongoengine (1/2)MongoEngineis aDocument-Object Mapper (think ORM, but fordocumentdatabases) forworkingwithMongoDBfrom Python. Ituses a simple declarative API, similartotheDjango ORM.http://mongoengine.org/
- 61. Mongokit (2/2)classBlogPost(Document): title = StringField(required=True)body = StringField()author = ReferenceField(User)created_at = DateTimeField(required=True) tags = ListField(StringField())blog_post = BlogPost(title='myblogpost', created_at=datetime.datetime.utcnow())blog_post.save()
- 62. MongoAlchemy (1/2)MongoAlchemyis alayer on top ofthe Python MongoDBdriverwhichadds client-sideschemadefinitions, an easiertoworkwithandprogrammaticquerylanguage, and a Document-Objectmapperwhichallowspythonobjectstobesavedandloadedintothedatabase in a type-safe way.An explicit goalofthisprojectistobeabletoperformasmanyoperationsaspossiblewithouthavingtoperform a load/save cyclesincedoing so isbothsignificantlyslowerandmorelikelytocausedataloss.http://mongoalchemy.org/
- 63. MongoAlchemy(2/2)frommongoalchemy.documentimportDocument, DocumentFieldfrommongoalchemy.fieldsimport *fromdatetimeimportdatetimefrompprintimportpprintclassEvent(Document):name = StringField()children = ListField(DocumentField('Event'))begin = DateTimeField() end = DateTimeField()def __init__(self, name, parent=None):Document.__init__(self, name=name)self.children = []ifparent != None:parent.children.append(self)
- 64.
- 65.
- 66. ACID versus BaseAtomicityConsistencyIsolationDurabilityBasicallyAvailableSoftstateEventuallyconsistent