Cargo Credible

1. Introduction

Overview:

Cargo Credible is a platform that aggregates data on over 3 million companies in the logistics space using Carrier-OK's API. We built business logic on top of this data that enables users to rate and review these companies, similar to Trustpilot.

The app allows users to search for a company by Name, MC, or DOT numbers directly from the homepage. Once a company is found, users are redirected to a search results page where they can view details such as contacts, phone numbers, addresses, DBA, insurance, compliance status, and equipment. Additionally, users can leave reviews on company profiles, providing industry-wide transparency and credibility.

Challenge:

Publicly available data on logistics companies is fragmented, difficult to access, and lacks transparency. Key challenges in building Cargo Credible included:

• Optimizing data storage, indexing, and querying across a dataset of over 3 million records.
• Ensuring compliance with Terms of Service (TOS) and privacy policies for data usage.
• Aggregating and scraping data to enrich company records, including associated resources such as nearby warehouses and legal services for truckers.

2. The Problem

Background:

In the logistics industry, brokers, carriers, and shippers struggle to access accurate, up-to-date, and transparent data on potential business partners. This lack of information leads to inefficiencies, trust issues, and operational risks. While there are some databases available, they often lack user-generated reviews or aggregated insights.

Pain Points:

• Data Accessibility: Information on logistics companies is spread across multiple sources, making it difficult to find consolidated insights.
• Performance & Scalability: Searching and retrieving records from a dataset of over 3 million entries required an optimized backend architecture.
• Regulatory Compliance: Aggregating and displaying public company data must align with legal and privacy regulations.
• Resource Aggregation: Truckers and logistics professionals need easy access to nearby warehouses, lawyers, and service providers.

3. Our Solution

Discovery Process:

We conducted an extensive audit of existing logistics databases and APIs, identifying key data points necessary for a comprehensive platform. Our approach prioritized efficient data storage, search performance, compliance with data policies, and an intuitive user experience.

Proposed Solution:

• Efficient Data Storage & Indexing:

  • Optimized data ingestion using PostgreSQL and Redis for caching frequently accessed records.
  • Indexed company records based on MC and DOT numbers for rapid search retrieval.

• Carrier-OK API Integration:

  • Integrated Carrier-OK's API to aggregate information on 3M+ logistics companies.
  • Applied business logic to normalize and structure data for better searchability.

• Review & Rating System:

  • Built a user-driven rating system for companies, similar to Trustpilot, allowing brokers and carriers to leave transparent feedback.

• Regulatory Compliance & Privacy:

  • Implemented robust Terms of Service and Privacy Policy guidelines to ensure data usage aligns with industry regulations.

• Resource Aggregation for Truckers:

  • Enhanced application with additional resources, including nearby warehouses and legal services for truckers.

Technology Stack:

• Backend: Django framework with PostgreSQL for structured data storage and Redis for caching.
• Frontend: HTML, JavaScript, and Tailwind CSS for a responsive user interface.
• Data Management: Carrier-OK API for company data aggregation, web scraping for supplementary resources.
• Infrastructure: Optimized search functionality using indexed queries and caching mechanisms.

4. Implementation

Data Storage & Search Optimization:

• Indexed company records by MC and DOT numbers for fast search retrieval.
• Used Redis to cache high-frequency searches and improve response times.
• Implemented background tasks for periodic data updates to keep records fresh.

Review System Development:

• Designed a structured review system allowing users to rate and comment on companies.
• Implemented moderation features to prevent spam and ensure credibility.

Compliance & Security Measures:

• Developed clear TOS and Privacy Policy guidelines to govern data usage with guidance from our legal partners.

Resource Aggregation:

• Integrated external data sources to provide additional business insights.
• Mapped nearby service providers such as warehouses and legal firms for truckers.

Challenges Encountered:

• Managing efficient queries across millions of records while maintaining low-latency responses.
• Balancing openness with compliance when displaying aggregated company data.
• Ensuring search accuracy with varying data formats and potential inconsistencies.

5. Results

Product Outcomes:

• A robust logistics database with fast and efficient search capabilities.
• A transparent rating and review system for logistics companies.
• Aggregated supplementary resources for truckers and logistics professionals.

Business Impact:

• Enhanced Transparency: Brokers and carriers can make informed decisions based on aggregated data and peer reviews.
• Optimized Performance: Indexed searches and Redis caching reduced query response times significantly.
• Scalable Architecture: Built a future-proof system that can accommodate further data expansions and integrations.

6. Lessons Learned

Key Takeaways:

• Performance Optimization Matters: Efficient indexing and caching significantly improve user experience in large-scale databases.
• User-Generated Content Adds Value: Transparent reviews and ratings help establish trust and credibility in the logistics industry.

7. Conclusion

Summary:

Cargo Credible successfully built an industry-first platform that aggregates logistics company data while allowing user-generated ratings and reviews. By integrating Carrier-OK’s API, optimizing search performance, and ensuring compliance, we created a scalable and transparent solution for the logistics industry.

Call to Action:

If you need a custom data aggregation platform, performance-optimized search solutions, or scalable web applications, let’s talk! Click the "Let’s Talk" button to get started.