Heading to NYC to discuss consumer health information exchange with @Medyears

Today I am on a flying visit to New York City to discuss a consumer-mediated Health Information Initiative Workshop in Philadelphia. We are planning this for early September.

The objective is to drive adoption of BlueButton and give patients easier access to their health data.

If we run this as a Hybrid HealthCa.mp event who is interested in getting involved?

More information to follow soon as we finalize details…

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1rOSdSP

#mongodbworld Elliott Horowitz talking about the future roadmap for @mongoDB

The Future of MongoDB. Beyond the next release.

Elliot Horowitz recapped:

Mongo 2.8 later in 2014.
- Improved Concurrency
- Storage Engine API

MongoDB 3.0 and Beyond

Partitioned Joins

Why no joins – Less need in a document relational model.
Also don’t want features that create surprise when scaling horizontally.

Multi-Document Transactions

Approach the same way as Partitioned Joins. Plan ahead on joins across collections to see that everything is on the same shard.

Schema Validation

For example add a Query document for a collection. Validate on submission against the Query document.

Multi-Master Databases

Incrementing counters for example can work in a multi-master environment

Filtered Replica Sets

eg. a Retailer might want data in every store.

Filtering means that only a section of the data is replicated. Eg. The UK Data Center gets UK data from a Global Data Center.

Storage Engines

The ability to tailor storage to performance needs. I think Health care will be big on having encryption as a storage engine.

Resource Management

Providing the ability to manage operations across different types of machines. This may be built in to future versions of MMS.

Adaptive Provisioning

MMS should be able to automatically adjust cluster size. This looks like some of the Object Rocket features will be embedded in to the core MongoDB offering.

Queryable Backups

MMS – Providing the ability to find a restore one or more documents without having to restore the entire database.
Connect the backup to a Mongod daemon and it would then be available to query.

Database as a Service Software

MMS will provide this capability for internal or external / cloud use.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1qbCijp

#MongoDBworld Hardware provisioning of @MongoDb – What I need to know!

Session details: http://2.healthca.mp/1lVpMCL

Hardware Provisioning for MongoDB

Some of the most common questions we hear from users relate to capacity planning and hardware choices. How many replicas do I need? Should I consider sharding right away? How much RAM will I need for my working set? SSD or HDD? No one likes spending a lot of cash on hardware and cloud bills can just be as painful. MongoDB is different from traditional RDBMSs in its resource management, so you need to be mindful when deciding on the cluster layout and hardware. In this talk we will review the factors that drive the capacity requirements: volume of queries, access patterns, indexing, working set size, among others. Attendees will gain additional insight as we go through a few real-world scenarios, as experienced with MongoDB Inc customers, and come up with their ideal cluster layout and hardware.

Chad Tindel

Senior Solution Architect at MongoDB

Chad Tindel is a Senior Solution Architect at MongoDB where he specializes in helping customers understand and use the nosql product to solve complex business problems. Previously, Chad was a Solution Architect at Cloudera focusing on the Hadoop space and was also a Solution Architect at Red Hat, helping customers build out their enterprise Linux infrastructures. He holds a BS in Computer Science from California Polytechnic in San Luis Obispo as well as an MS in Finance from the University of Denver.

Hardware Provisioning

There is not a lot of information out there on sizing MongoDB. This session, even though the last session was well attended.

How do you size?

Often customers over or under engineer.

Think the scenario where your app gets listed and suddenly lots of sign ups. The server gets beaten and needs to be re-sized.

Requirements – Step 1

What are the business requirements

• Uptime (do you need more than one Data Center
• Availability
• Throughput
• Responsiveness
• Acceptable latency – especially during peak times.

Constraints

Resources available

Continuing Requirements

• Requirements can change over time
• More users, more data, new indexes
• More writes

Guidance:

• Collect metrics!
• Adjust configuration incrementally
• Plan ahead

Try to avoid a crisis.

Do a Proof of Concept

• Start small on a single node
• Design your schema (Read and write applications are different)
• Understand query patterns
• Get a handle on working set (the active data)

Then add replication to see impact

Review Requirements as result of POC

• Data sizes (Number of documents, Average document size, size of data on disk, size of indexes, expected growth, document model)
• Ingestion – Throughput / Updates / Deletes per second peak and average
• Bulk inserts? How large and How often?

Do you have SLAs on this performance?

• Performance expectations
• Life of data
• Security requirements (SSL, Encryption at rest)
• Number of data centers in use (Active/Active , Active/Passive Cross Data Center latency)

Resource Usage:

IOPS (4K in size)
Size
Data and loading patterns.

CPU tends to be less important

Fast storage and as much RAM as you can.

Network latency affects replication lag

IOPS

7200 RPM SATA = 75-100 IOPS
15000 SAS = 175-210 IOPS
Amazon SSD EBS = 4,000 PIOPS / Volume
48,000 PIOPS / Instance

Intel X-25-E SLC = 5,000 IOPS

Use IOSTAT to monitor disk performance (or MongoPerf).

Release 2.4 added a feature to estimate the size of a working set.

Network Performance

Latency impacts WriteConcern time and ReadPreference

Throughput impacts Update and Write Patterns and Read/Queries

Use Netperf to measure network performance.

CPU Usage

Only really comes in to play when using queries without indexes which mean performing a table scan.
or for Sorting within a Shard and MergeSorted when aggregated.

Aggregation Framework or MapReduce require CPU Performance.

Case Study – Spanish Bank:

• 6 months of logs held for 6 months
• 18TB at 3TB/Month

• Prod environment:
  3 Nodes / shard * 36 Shards = 108 Physical Machines
  128GB/RAM * 36 = 4.6TB RAM

2 Mongos
3 Config servers (virtual machines)

Online Retailer

• moving Product catalog from SQL Server to MongoDB as an overhaul to Open Source
• 2 Main Data Centers active/active
• Cyber Monday peaks at 214 Requests/Sec. Budget for 400 Requests/Sec for headroom.
• Heavy Read process orientation.

POC

• 4 M product SKU’s with JSON document size of 30KB
• Requests for specific product (by _id)
• Products by Category (Return 72 documents – or 200 if a google bot)

Math

• Partition (Shard) by Category.
• Products in multiple categories are duplicated means on average doc is in 2 categories so store 4M SKUs x 2 = 8M

8M docs * 30K want everything in memory. 384GB RAM/Server

Sharding adds a layer of complexity (eg. Add config server) so don’t shard unless you need to.

Determined a 4 Node Replica set 2 in each Data Center. Plus an Arbiter.

Recommended a Single Replica Set
- 4 Node Replica

But customer found they could only deploy on 64G RAM. So they deployed 3 shards 4 nodes each + Arbiter.

Arbiters are small. They just exist for voting. Can be a small 1VCPU with 4GB RAM.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1mfQzDU

#MongoDBWorld @Genentech: Speeding Drug Research

This is my third session looking at the use of MongoDB in a health setting. See my earlier posts from today:

• MongoDB at the Veterans Administration
• Translational Medicine and MongoDB at Sanofi

The Genentech session information:

The Best of Both Worlds: Speeding Up Drug Research with MongoDB & Oracle (Genentech)

Genentech Research and Early Development (gRED) develops drugs for significant unmet medical needs. Key to this effort is providing Investigators with new genetic strains of animals needed to understand disease causes and test new drugs. While these genetic strains have increased greatly in complexity, technology improvements have increased accuracy and throughput while reducing the cost of genetic testing. This has led to an effort to redevelop the Genetic Analysis Lab system to reduce the time needed to introduce new lab instruments from months to weeks or even days. Important to this initiative has been the introduction of MongoDB to capture the variety of data generated by genetic tests and integrate it with the existing Oracle RDBMS environment. Not only has it proved fairly easy to integrate the two, but we have been able to take advantage of the strengths of MongoDB to provide a flexible schema and Oracle to provide transaction management and integration with the existing information system.

Doug Garrett

Software Engineer in Research at Genentech

Doug Garrett, Software Engineer in Research at Genentech, has been developing software for over 20 years. Most recently he has worked on developing systems to support the processes and lab instruments needed for the development of genetic murine models needed for Genentech disease and drug research. Before Genentech, Doug worked for a number of different companies including Nokia, McKesson and Kaiser Medical. Doug holds a B.A. in Physics from Occidental College and an M.S. in Computer Information Systems from Boston University.

Genentech – Speeding Drug Research

The challenge was to integrate MongoDB with Oracle Relational Databases.

BioInformatics is different from IT.

• The flexibility of the schema is a big benefit
• It can also easily integrate with traditional RDBMS
• Saving time is critical when you are dealing with saving human life.

Every new lab instrument drove a change to the Oracle RDBMS schema. This created a time lag and slowed genetic testing.

The Development process

• What is a disease cause? Is it genetic
• Develop new mouse model
• Does it create a new drug.I s it safe and effective
• Then move to clinical trials

With Oracle and their schema it took 6 months to modify the schema to add a new genetic test. The follow on additional test took a further 3 months. A more flexible solution was needed that didn’t add to the complexity of the database.

This led to the selection of MongoDB.

1 million rows in Oracle became 4,000 documents in MongoDB.

The ingestion process is where the configuration for an instrument is focused.

There is then a generic data loader. The schema complexity is in the mongo document and maintained in one place.

A java program presents users with a single window. This combines the Oracle record view with the content of the relevant MongoDB document(s).

The DB Schema is now immune from introduction of new instruments.

Going Live

Issues – A Disaster Recovery copy

Oracle replication is challenging. It is not a built in function

With MongoDB a replica was implemented in the DR site within a couple of hours.

MongoDB Aggregation Framework change in Release 2.6 the 16MB limit on result sets was removed.

Other uses for MongoDB

Import from CSV, JSON, XML and other sources.
MongoDB as a data import service great for building data pipelines.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1sF9hhN

#mongoDBWorld @Sanofi talking about Cancer research Translational Medicine

Sanofi – Big Data and Translational Medicine

Session information: http://2.healthca.mp/1ryzDOj

Presentation by David Peyruc and David Erwan

www.sanofi.com

third largest Pharma company

Invest 4.7B Euros per year. From revenue of 33.4B Euros.

The challenge for Pharma

The classic business model is under threat. Generic drugs are a threat – “The Patent Cliff”

End of the Blockbuster Age.

New Paradigms

• Personalized
• Predictive
• Preventive
• Participatory

Translational Medicine is about bridging the gap between Clinical and Research worlds. Linking Hypothesis with Evidence.

Genomics and other *omics data needs to merge with other data. (See my other post from the Broad Institute from yesterday).

Translational Medicine Challenges

• Diversity of data objects.
• Large storage requirements. Genomics data is big.
• Consistency and Traceability
• User Friendly curation process for annotation to enable understanding and extract knowledge

Big Data (MongoDB) is used to help Extract, Curate, Normalize and Load data.
MongoDB is the central repository for Biomarker data.

Why MongoDB?

• File and Metadata together.
• Scalable (Data was sharded from day 1)
• Easy to instal, use,understand and adapt/adopt.

The Journey to MongoDB

• Big Data White Paper
• Install and Benchmark
• Proof of concept (a few javascript pages to demonstrate access)

MongoDB
- Runnong GridFS
- Apache Solr for search access
- Collections of Metadata, Config, User profile, Logs, tools etc.

• Implement a REST API service layer
• Build a web, Desktop and third party software integration interface (using Rest API)

Use Cases

360 degree Explorer

• Disease or Syndrome / Receptor

Geographic Zone / Health Activity

• Show the same data via faceted navigation

The platform is not a standard IT product. So there is minimal IT support.

Benefits

Scientists benefit:

• More efficient tagging and curation
• Awareness of more data sources
• Easy exploration of data
• Easier integration of external data

IT

• Faster development
• Flexibility
• Performance
• Documentation, support, training and community support.

Twitter connections for the presenters.
@bob_dit_l_ane
@DPeyruc

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1q9PAN8

#mongodbworld Werner Vogels CTO from @Amazon – #IOT the Internet of Things is already here

http://2.healthca.mp/1pB0jfP

Werner Vogels – The Internet of Things is here

Great presentation from Werner

“The amount of information recorded about a baby in the first day of life is 70 times the information contained in the Library of Congress”

Observations – Theory – Models – Facts

The mapping of the Human Genome. The first genome map cost billions of dollars. Now the 1,000 Genome Project at NIH. 1700 Genomes are freely available on AWS. It takes about 200TB.

Illumina has BaseSpace – Analytics for Genomics.

Unilever is doing deep sequenced Genome analytics.

The Ocean Observatories have deposited sensors on the ocean floors and are pulling data in to AWS for analysis.

The Mars Rover is using AWS S3 to merge 2 Mega Pixel camera images to create large panoramas.

Consumer world

Dropcam – just got purchased by NEST/Google for $550M. Providing a security camera feed. Dropcam is one of the biggest inbound video service on the web. More data uploaded than YouTube. Petabytes per day.

[Ed: I wonder how soon this will go to Google's App Engine?]
.
Glowcap / GlowPack -

If you don’t take your meds on time the caps glow. If you ignore it plays a tune. If you keep ignoring it notifies your social network.

DirectLine in the UK – Insurance Company

Capturing data from your smartphone while you drive. Feeds your driving profile to the insurer. Drive well get a discount!

Ask NAO

Autistic kids are happier talking to robots than to people.
So Robots interact with Autistic kids.

Retail

Analytics have always been popular in retail Who is my customer? What are they saying/doing?

All this drives to Personalization.
The More data you can collect the more accurate your recommendations can be.

Werner gave some great examples where the Amazon recommendation fails – because there is not enough data.

Industrial

GE Engines – Turbines are being instrumented. The data flows to AWS and is analyzed. Efficincies of 1% can generate millions of dollars of savings.

Shell have instrumented their oil wells.

KArcher is a german industrial cleaning company. All the devices they build are data generators. The data is fed back to the equipment fleet owners.

DeConstruction – Built mBuilder sensors for building sites. The data flows in to an analytics dashboard to tell construction managers what is happening on a building site.

Sports

Each professional sports team has an analytics expert that can influence gameplay.

Some teams are equipping players with heart monitors. Players with same heartbeat profile are in sync and the team performs better.

Shockbox – Developed a strip for kids Ice Hockey helmets to assess likelihood of concussion.

Forusquare A big ongoDB user: – Social Cooler – The beer cooler only opens up if you check in with 3 of your friends. No drinking alone!

Tata – Predictive monitoring for Preventative Maintenance of Truck fleets.

OneBusAway – Public Transit app. Instrumented buses.

Waze or Moovit: Waze for private transport user generated content. Moovit is the same for public transport – Both are isreali companies. Waze was acquired by Google

Going real time.

We don’t want to know what happened yesterday. We want to know what is happening NOW.

Amazon built a real time tool. Amazon Kinesis

Managed service for real-time processing. Data is streamed in, sequenced and then output to storage eg. S3, DynamoDB etc.

Beyond the Display

To change behavior you have to find different interfaces.

Ambient Bus Pole – The light rises on the pole as the bus approaches.

Pre-Paid Electricity company provided customers with a lamp that glows when their account gets low. This solved people failing to top up their accounts on time.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1pB0jfN

#MongoDBWorld United Health Group using @MongoDB to handle NPPES (Provider) data

More from #MongoDBWorld.

How UnitedHealth Group Integrated Open Data into its MongoDB-Based Information System

http://2.healthca.mp/VgNI8R

Cedric Cabone, CTO of Talend and Matt Axsom, Lead Architect at Optum – United Health Group

Talend is a unified platform, a code generator with distributed architecture. Open source.

Matt Axsom is with Optum. They are focused on Clinical Performance and Compliance handling Encounter data processing.

Optum use Talend Studio for process automation, orchestration and execution for business continuity and ROI.

Enable business users to change their analytics.

RADAR – Risk Adjustment Data Acquisition and Analysis Repository

Medicare and Retirement Claims related data.
EHR/EMR/HL7 data.

Handling Trading partner file submissions (tracked via GridFS)
Transactions via Collections, correlated with Metadata
Downstream file outputs are stored in GridFS
Audit traceability from cradle to grave.

Simplified audit-ability an research.

Trading Partner Submissions

Hospitals and Providers and clearing houses.
Transactions include: Provider Direct, Hospital Data Capture, Chart Review, Delete Submissions

Trading Partner Submissions: Inputs are passed through Talend ETL, depositied in MongoDB then pased to external CMS (Medicare/Medicaid) and Internal RDBMS.

MongoDB collections: Trading Partner, GridFS, Encounters, Submissions.

The process started from:

Select * FROM TX_Detail where Date = “20140623″

Low priority but saves a business user 30 minutes per day to pull data and export to Excel and generate a status email.

Large datasets so not suitable for laptop/desktop.

This drove the selection of Talend. Talend does orchestration to SAS, SSIS and SAP (Business Objects/Crystal reports) web hooks (REST/SOAP APIs) as well as others.

No utility scripting required. Talend orchestrates across systems.

Talend can integrate with anything that is accessible from the command line.

MongoDB selection was driven by…

The file system didn’t have reference to objects. It made audit-ability difficult.

Alternatives were BFILE/HDFS/GridFS.

The operational aspects made the selection of MongoDB preferable. eg. Cloning data for development staging, needing to run parallel data streams in development.

Data Intake Issues

A lot of disparate data formats. Many can’t be changed.

RDMS Challenges

Modeling of data in RDMS was complex due to data disparity. Building a schema to account for all data streams gets massively complex.

Use of Hadoop

• YARN provides Cluster Resource Management
• Talen can create Pig latin script

• HBASE requires taking system down for upgrades.

• MongoDB offers operations stability. It has only been taken down one in 21 months.
• HBase only came to Production level in Q1 2014. Whereas Optum had been running since October 2012.

Operational simplicity wins out.

Secondary indexing was required by Optum but this was not available until December 2012. Prior to that an equvalence to secondary indexing required a full table scan.

Existing environment:
- 20 Table Join
- 4 billion Rows
- 856K row input driver
- 100 hour execution cycle

Hive Queries are seeing a 100 to 1 improvement by using MongoDB.

OLTP Platform load to Hadoop
- File to HDFS
- Sqoop to HDFS/Hive

Talend:
Hive 92SQLish
Pig – Transformation
Map-Reduce (Data Read . Shuffle and Sort) Low level Java
HCatalog* – Shared Schema / Data Typing
Partition Notifications on load – MQ Style hook to trigger subsequent analytics after load

SAS
- Hive

Tableau / SAP Business Objects 4
- Hive

Current State:

100 daily jobs
Moving/Migrating jobs to Data Services
Leveraging tMOM and Camel
Big Data
- Migrating all core ODS to MongoDB
- ETL/ELT Analytics offload to Hadoop where it makes sense.

Looking at MongoDB Hadoop connector to allow direct run from Hadoop.

NPI Engine:

NPI – National Provider Identifier

Monthly file is approx 4GB in size.
4+ Million Records
329 Columns

publicly available data updated frequently.

NPI Permissible use:

Fraud waste and abuse
Patient Provider linkage.

It is the single source of truth.

PECOS from Government has additional data.
Augment with internal provider data

OIG – list of excluded individuals and entities
SAM – System for Award Management

Acceptable Physician speciality type.

Need to mashup these government data sets.

Social Services Administration produces Death Master – Who has died.

I think I want to look at use of RestCAT Open Source API to ingest MPPES in to MongoDB.

CMS captures Licensure but DOES NOT Validate that data. Taxonomy data may be questionable but NPI data is consistent.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1v2k5SS

#MongoDBWorld Learning about @LinkedIn Learning with @MongoDB

More from #MongoDBWorld.

LinkedIn uses MongoDB to quickly create APIs.

“What’s your next play?”

LinkedIn developed “LearnIn” “Transform yourself through Learning”

Find resources when you need it.

Want employees to search across different platforms. eg. Safari books, training course, internal wiki etc.

Also wanted to create a living breathing transformation plan – used for personal development.

What technologies to use?

Primarily a javascript team.

What server and database to use?

How to build it quickly?

Server selection

No engineering support. So as a javascript team they set out to use Node.js.

• Server-side Javascript
• Lightweight
• NPM Package Support
• Extensive support in the community
• Allows for data-driven JSON

Which database?

The Needs:
- support 5,000 employees, globally
- Minimal data storage needs
- More reads than writes
- Scale with company growth
- Flexible

In past year company has grown from 3,000 to 6,000+ – Hyper growth

The database choice was MongoDB because:

• Easy to setup NoSQL Joson
• Node.js driver
• Advanced queries
• Flexible Schema – schema-less
• Extensive documentation
• High single instance threshold (single instance can hold Terabytes of data)

Taking schema-less approach to MongoDB you can break API/Application logic.

They went to Mongoose – An Object Document mapper. Mongoosejs.com

• Easy type casting
• Quick setup
• Easy Document modeling
• field validation API
• Business Logic hooks, Custom Middleware
• Mongo_id reference population

Mongoose forces I collection per data type

This helped with:
- Normalized data modeling
- Many-to-many relationships
- Collection per Data Type

How do we search?

How to search on documents via text search.

MongoDB provides full-text search (beta in v2.4)

• Easy searching via a MongoDB index
• Relevancy Searching
• Stemming and multi-language support.

Some limitations:
- Single token/analyzer.
- Simple relevancy scoring but not Lucene
- No completion suggestion
- No fuzzy matching
- No related item search.

What was needed:
- Lucene index with relevancy scoring and performance
- Custom field analyzer for tokenization and stemming
- “Related to” querying or “More Like This”
- Quick completion suggesiton
- Complicated wildcard search
- Easy Node.js integration

Went to ElasticSearch (elasticsearch.org)

Getting ElasticSearch to work with MongoDB.
elasticseach-river-mongodb

River takes a stream of data and indexes in realtime.

River plugs in to the MongoDB Oplog. and creates elasticsearch index using same opLog that is used to create MongoDB Replicas.

elasticsearch.js can plug in to node.js api.

The limitations of elasticsearch:

• keeping index clean requires data clean up
• Adds more technology to the stack

Questions

What IDE does the team work?

• Webstorm

Do you lose flexibility with Mongoose?

• Not really. It does make you think about data structures but you can easily change them.

Jacob Dejno @dejno – Web Developer
Ryan Seamons @ryanseamons – Product Manager LearnIn

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1v26oDz

#MongoDBWorld Learning about @LinkedIn Learning with @MongoDB

More from #MongoDBWorld.

LinkedIn uses MongoDB to quickly create APIs.

“What’s your next play?”

LinkedIn developed “LearnIn” “Transform yourself through Learning”

Find resources when you need it.

Want employees to search across different platforms. eg. Safari books, training course, internal wiki etc.

Also wanted to create a living breathing transformation plan – used for personal development.

What technologies to use?

Primarily a javascript team.

What server and database to use?

How to build it quickly?

Server selection

No engineering support. So as a javascript team they set out to use Node.js.

• Server-side Javascript
• Lightweight
• NPM Package Support
• Extensive support in the community
• Allows for data-driven JSON

Which database?

The Needs:
- support 5,000 employees, globally
- Minimal data storage needs
- More reads than writes
- Scale with company growth
- Flexible

In past year company has grown from 3,000 to 6,000+ – Hyper growth

The database choice was MongoDB because:

• Easy to setup NoSQL Joson
• Node.js driver
• Advanced queries
• Flexible Schema – schema-less
• Extensive documentation
• High single instance threshold (single instance can hold Terabytes of data)

Taking schema-less approach to MongoDB you can break API/Application logic.

They went to Mongoose – An Object Document mapper. Mongoosejs.com

• Easy type casting
• Quick setup
• Easy Document modeling
• field validation API
• Business Logic hooks, Custom Middleware
• Mongo_id reference population

Mongoose forces I collection per data type

This helped with:
- Normalized data modeling
- Many-to-many relationships
- Collection per Data Type

How do we search?

How to search on documents via text search.

MongoDB provides full-text search (beta in v2.4)

• Easy searching via a MongoDB index
• Relevancy Searching
• Stemming and multi-language support.

Some limitations:
- Single token/analyzer.
- Simple relevancy scoring but not Lucene
- No completion suggestion
- No fuzzy matching
- No related item search.

What was needed:
- Lucene index with relevancy scoring and performance
- Custom field analyzer for tokenization and stemming
- “Related to” querying or “More Like This”
- Quick completion suggesiton
- Complicated wildcard search
- Easy Node.js integration

Went to ElasticSearch (elasticsearch.org)

Getting ElasticSearch to work with MongoDB.
elasticseach-river-mongodb

River takes a stream of data and indexes in realtime.

River plugs in to the MongoDB Oplog. and creates elasticsearch index using same opLog that is used to create MongoDB Replicas.

elasticsearch.js can plug in to node.js api.

The limitations of elasticsearch:

• keeping index clean requires data clean up
• Adds more technology to the stack

Questions

What IDE does the team work?

• Webstorm

Do you lose flexibility with Mongoose?

• Not really. It does make you think about data structures but you can easily change them.

Jacob Dejno @dejno – Web Developer
Ryan Seamons @ryanseamons – Product Manager LearnIn

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1v26n2l

#MongoDBworld @MikeOlson of Cloudera – A new way to think about data

More from #MongoDBWorld.

Mike Olson – Founder and Chief Strategy Officer – Cloudera @mikeolson

“Machines are much better at generating data than people.”

Data is growing faster than Moore’s Law

The interactions of devices makes this growth exponential.

Classic data management is old, big, centralized and expensive. This is not good for machine data.

Google developed tools for handling machine generated data.
Amazon has promoted the cloud. They are one of the most innovative companies.
The Cloud is fundamental. It is a participant, a generator and a recipient.

Data will be generated in lots of places and will live where it was born.

The Data Center of the future will span firewalls.

Book Recommendations: Small Pieces Loosely Joined: A Unified Theory Of The Web – Dave Weinberger.

Nonzero: The Logic of Human Destiny – Robert Wright

Enterprise Data Hub

The initial idea was one system to rule them all but Cloudera has discovered that this doesn’t work. Systems have to talk to other systems. Systems loosely joined for a non-zero sum game.

Example: Terradata, MongoDB and Cloudera have created an enterprise data fabric that stimulates new value.

Collaboration and Connections will win out.

Disruption

New obliterates old. After the hype they co-exist and are targeted for purpose.

Disruption is not the future. Transformation is the future.
Using Data in ways we hadn’t predicted.

“The future of data is that data is the future. It will infuse how we do everything.”

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/V9Chj7

#MongoDBWorld kicks off

I am here at the Sheraton in Times Square New York for MongoDBWorld 2014. I will be blogging from the conference. There are some great topics. I am particularly interested in some of the sessions on Genetics and the NPI data source.

The conference runs for the next two days (June 24-25th) and is a heavily packed agenda. See the full schedule here: http://2.healthca.mp/1p9VQF7

There are around 2,000 people here for this event from across the world.

It is going to be a busy day because I also have to fit in meetings for Medyear.

# Max Shireson – Kenyote @mshireson

The Last 44 years of Data was all neat and Tidy

The changing world:

PC to Mobile
Ads to Social

Data is different today. 90% of world’s data is less than 2 years old.
80% of enterprise data is unstructured.
Unstructured data growing at 2x the rate of structured data.

2/3 of Fortune 500 are challenged by unstructured data.

6*% Data Variety
15% Data Volume
17% Other Data

A decade ago Google emerged as one of most valuable companies. Took visible data and organized it. That spawned Big Table and other tools.
Open Source was critical because licensing software for the machines to index the Internet would kill any data model.

There have been 7M downloads of MongoDB
20,000 deployments

Major vendors are supporting MongoDB eg. IBM, Cloudera, SAP.

some examples of MongoDB users

Mailbox – Scaled to millions of users in weeks. Subsequently acquired by Dropbox.

Bosch - The Internet of Things will outnumber people on the Internet. They are building the infrastructure for the car. They don’t know what will be required in terms of data structures. MongoDB gives them the flexibility.

MetLife – Data in 70 different systems – Tried to pull togeth in Relational database. They couldn’t keep up with the change. MongoDB was implemented to integrate these data streams in a call center in 90 days.

City of Chicago – Starting from the CTO’s laptop grew to MongoDB with Hadoop to power emergency communications rooms and more critical projects across the city.

Citi – Building a shop window to deploy computers and clusters quickly.

## The future of MongoDB

Developing features: Concurrency, Storage and Management.

[tag health cloud BigData MongoDB MongoDBWorld NoSQL]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1v1zuD2

In NYC for #MongoDBWorld and for @Medyears meetings

This week I am in New York City to take part in MongoDB World. You can still grab yourself a ticket to the event on Tuesday and Wednesday at the Sheraton in Times Square. Check out the link here: http://2.healthca.mp/1wmC9tz You will have to be quick. Ticket sales close at 10am ET today (Monday 23rd).

Today is a series of training sessions at MongoDB HQ in NYC. Unfortunately I am not getting to go to those sessions but I will be blogging from the sessions on Tuesday and Wednesday.

I am also here to take part in a number of meetings for Medyear. There is a lot happening with Medyear. The ability to import BlueButton data and share it securely is attracting a lot of attention. Meanwhile we continue to refine both the iOS App and the web site. Our approach in which we break down your health timeline in to a series of posts that can be discretely and securely shared gives a lot of flexibility and using hashtags for public and private chronicles has created a rich platform that gives our members a simple set of tools that offer tremendous flexibility.

I keep finding new ways to build my own health timeline on Medyear.com. For example, I have been able to use IFTT.com to create workflows that pipe my Fitbit data to my Medyear timeline. We are working on relaunching our corporate presence as Medyear.org. That will include a blog and I want to publish some ideas about how people are/can use their Medyear.com account to gather their health data in to one place.

Anyone can sign up for a free account on Medyear.com and start aggregating their health data. If you do, let me know I would love to learn about how you are using Medyear. Medyear is totally patient/member-centric so we want to hear from you about what you need to help you manage your health data.

It is early on Monday morning. After fueling with some Caffeine after a 3am start I am heading to Blueprint Health until early afternoon. If you are in the SoHo area why don’t we connect.

[tag health cloud BigData MongoDB NoSQL MongoDBWorld]

Mark Scrimshire
Health & Cloud Technology Consultant

Mark is available for challenging assignments at the intersection of Health and Technology using Big Data, Mobile and Cloud Technologies. If you need help to move, or create, your health applications in the cloud let’s talk.
Blog: http://2.healthca.mp/1b61Q7M
email: mark@ekivemark.com
Stay up-to-date: Twitter @ekivemark
Disclosure: I began as a Patient Engagement Advisor and am now CTO to Personiform, Inc. and their Medyear.com platform. Medyear is a powerful free tool that helps you collect, organize and securely share health information, however you want. Manage your own health records today. Medyear: The Power Grid for your Health.

via WordPress http://2.healthca.mp/1wmC6xV