Generate a Newsletter with Exa Research Agent and CrewAI

$ pip install crewai

$ pip install crewai
$ crewai new newsletter-crew

$ poetry add my-dependency

research_task:
  description: >
    Conduct a thorough research about the latest news on {topic}. 
    Be sure to look for sources that are reliable and publish recent news.
    Do not include articles that are not news material or that are not 
    directly related to {topic}.
    
    With this research, compile a list of the most relevant news 
    stories that you found. 

    Follow these rules:
    - Only include articles that are especially relevant to {topic}. 
    Do not include any news that are not directly related to {topic}. 
    - Do not include sources that are not a news article. 
    If the content of the page includes a list of articles or 
    looks like the front page of a website, do not include it in the list!
    - Summarize the news in a few sentences. Make the summary as long 
    as necessary to include all the relevant information, but not too 
    long for a newsletter.
    - Include the URL of the article where you found the news.
    - Include a minimum of 7 news articles and a maximum of 10 news 
    articles in the list.
    - When using the Search Tool, your search query should be concise 
    (for example, "latest news on {topic}").

    IMPORTANT INSTRUCTIONS ABOUT USING TOOLS: When using tools, 
    DO NOT ESCAPE the underscore character "_", EVER. If you need 
    to use a tool and pass in a parameter called 'search_query', 
    you should write 'search_query', not 'search\_query'. 
    THIS IS VERY IMPORTANT, else the tool will not work.    

  expected_output: >
    A markdown document with the most relevant news stories. Each 
    news story should contain the following:
    - Title of the news
    - Summary of the news
    - URL of the article where the news was found

    Here is an example of the format of a news article that you 
    could include in the document:
    
    <EXAMPLE>
      Story 1:
      - Title: **Daily briefing: AI now beats humans at basic reading 
      and maths**
      - **Summary:** AI systems can now nearly match and sometimes exceed 
      human performance in basic tasks. The report discusses the need for new 
      benchmarks to assess AI capabilities and highlights the ethical 
      considerations for AI models.
      - **URL:** [Nature Article](https://www.nature.com/articles/d41586-024-01125-1)
    </EXAMPLE>    

edit_task:
  description: >
    Given the list of news articles that will be used in the newsletter, 
    do the following things:
    
    - Rewrite the title of each news article to make it more engaging and 
    interesting for the readers of the newsletter.
    - Add a paragraph to each news article that explains why this news is 
    important and how it can impact the readers of the newsletter.
    - Reorder the bullet points in a way that the most relevant news and 
    topics are at the top of the list based on the importance of the news 
    and topics.
    - Verify that the news articles are directly related to {topic} and that 
    they are not off-topic. If they are off-topic, remove them from the list.
    - Verify that the URLs are correct and that they lead to the correct news 
    article. They should lead to a news article and not to a list of articles 
    or the front page of a website. If the URL is incorrect, ask the researcher 
    to provide the correct URL.
    - Do not search for additional news articles or change the content of the 
    news articles. Only edit the existing news articles.

    IMPORTANT INSTRUCTIONS ABOUT USING TOOLS: When using tools, DO NOT ESCAPE
    the underscore character "_", EVER. If you need to use a tool and pass in
    a parameter called 'search_query', you should write 'search_query', not 
    'search\_query'. THIS IS VERY IMPORTANT, else the tool will not work.    

  expected_output: >
    A markdown document with all the news to be included in the newsletter 
    of the week. The document should have a title related to the curated 
    stories of the week and a list of news articles.
    
    Each one should contain:
    - Title:
    - Summary: 
    - Why this is important:
    - Source: [URL of the article]

    Here is an example of a document that you are expected to produce:
    <EXAMPLE>
    Title of the day: AI is taking over the world

      - **Title:** AI Surpasses Human Capabilities in Basic Reading and Maths
        **Summary:** Recent advancements in AI technology have enabled 
        systems to match and sometimes exceed human performance in fundamental 
        tasks such as reading and arithmetic. This shift necessitates the 
        creation of new benchmarks to appropriately evaluate AI capabilities.
        **Why this is important:** This development is crucial as it not only 
        showcases the rapid progress in AI but also prompts a reevaluation of
        how AI is integrated into educational and professional settings. It 
        raises significant ethical questions about the future role of AI in society.
        **Source:** "Title of the article" by ["Source"](https://www.nature.com/articles/d41586-024-01125-1)

        [... more news articles ...]

    </EXAMPLE>    

newsletter_task:
  description: >
    Fill the following HTML template with exactly the same information that 
    is given to you. Also, include the following personal message at the 
    beginning of the newsletter in the space provided: {personal_message}

    (If there is no personal message, leave the space empty.)
    
    Follow these rules:
    - Do not truncate the information or change the order of the elements. 
    - Do not modify the content of the news articles.
    - Do not add any additional information to the newsletter.
    - Add ALL the news stories provided to the newsletter. Even if the 
    template only shows one news story, you should include all the news 
    stories provided as context using the format provided in the template.

    {html_template}    
  expected_output: >
    Return ONLY the contents of the HTML template, without any tripple 
    quotes like '```', '```html' or any other text. 
    This output will be saved as a HTML file and sent to the subscribers.    

researcher:
  role: >
        Senior Researcher
  goal: >
        Uncover cutting-edge developments in {topic}.
  backstory: >
    You're a seasoned journalist with a nose for news. You're known for your great research skills and ability to dig up the most interesting stories. Your reports are always thorough and well-researched, making you a trusted source of information.
    You always follow the rules and guidelines provided to you and you never forget to include the complete URL of the article where you found the news.    

editor:
  role: >
        Editor-in-Chief
  goal: >
        Ensure the quality and accuracy of the final newsletter.
  backstory: >
    You are the Editor-in-Chief of a prestigious news organization. You are responsible for overseeing the production of the newsletter and ensuring that it meets the highest standards of quality, that it is accurate, well-written, and engaging. 
    You review the news articles provided by the researcher, add context to each article (like why the news story is relevant), and have a great sense of what will resonate with the readers. You use this sense of judgment to reorder the news articles in a way that the most important news is at the top of the list.    

designer:
  role: >
        Newsletter Compiler
  goal: >
        Fill the HTML template given to you with the news articles provided.
  backstory: >
    You are responsible for compiling the HTML code of the newsletter, making sure that every news article is included in the final document.
    You do NOT modify the content and only update the design when necessary. You use the HTML template provided to you to create the newsletter.    

from crewai_tools import BaseTool
from exa_py import Exa
from datetime import datetime, timedelta
import os


class Search(BaseTool):
    name: str = "Search Tool"
    description: str = (
        "Searches the web based on a search query. Results are only from the last week. Uses the Exa API."
    )

    def _run(self, search_query: str) -> str:

        one_week_ago = datetime.now() - timedelta(days=7)
        date_cutoff = one_week_ago.strftime("%Y-%m-%d")

        exa = Exa(os.getenv("EXA_API_KEY"))

        search_response = exa.search_and_contents(
            search_query,
            use_autoprompt=True,
            start_published_date=date_cutoff,
            text={"include_html_tags": False, "max_characters": 8000},
        )

        return search_response


class FindSimilar(BaseTool):
    name: str = "Find Similar Tool"
    description: str = (
        "Searches for similar articles to a given article using the Exa API. Takes in a URL of the article."
    )

    def _run(self, url: str) -> str:

        one_week_ago = datetime.now() - timedelta(days=4)
        date_cutoff = one_week_ago.strftime("%Y-%m-%d")

        exa = Exa(os.getenv("EXA_API_KEY"))

        search_response = exa.find_similar(url, start_published_date=date_cutoff)

        return search_response


class GetContents(BaseTool):
    name: str = "Get Contents Tool"
    description: str = (
        "Gets the contents of a specific article using the Exa API. Takes in the ID of the article in a list, like this: ['https://www.cnbc.com/2024/04/18/my-news-story']."
    )

    def _run(self, ids: str) -> str:
        exa = Exa(os.getenv("EXA_API_KEY"))
        contents_response = exa.get_contents(ids)
        return contents_response

from crewai import Agent, Crew, Process, Task
from crewai.project import CrewBase, agent, crew, task
from typing import Dict, List, Tuple, Union
from langchain_core.agents import AgentFinish
from langchain_openai import ChatOpenAI

from newsletter_gen.tools.research import Search, FindSimilar, GetContents

import datetime
import json
import os

@CrewBase
class NewsletterGenCrew:
    """NewsletterGen crew"""

    agents_config = "config/agents.yaml"
    tasks_config = "config/tasks.yaml"

    def llm(self):
        llm = ChatOpenAI(api_key=os.environ['OPENAI_API_KEY'], model='gpt-4-turbo')
        return llm

    @agent
    def researcher(self) -> Agent:
        return Agent(
            config=self.agents_config["researcher"],
            tools=[Search(), FindSimilar(), GetContents()],
            verbose=True,
            llm=self.llm(),
        )

    @agent
    def editor(self) -> Agent:
        return Agent(
            config=self.agents_config["editor"],
            verbose=True,
            tools=[Search(), FindSimilar(), GetContents()],
            llm=self.llm(),
        )

    @agent
    def designer(self) -> Agent:
        return Agent(
            config=self.agents_config["designer"],
            verbose=True,
            llm=self.llm(),
            allow_delegation=False,
        )

    @task
    def research_task(self) -> Task:
        return Task(
            config=self.tasks_config["research_task"],
            agent=self.researcher(),
            output_file=f'logs/{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M")}_research_task.md',
        )

    @task
    def edit_task(self) -> Task:
        return Task(
            config=self.tasks_config["edit_task"],
            agent=self.editor(),
            output_file=f'logs/{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M")}_edit_task.md',
        )

    @task
    def newsletter_task(self) -> Task:
        return Task(
            config=self.tasks_config["newsletter_task"],
            agent=self.designer(),
            output_file=f'logs/{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M")}_newsletter.html',
        )

    @crew
    def crew(self) -> Crew:
        """Creates the NewsletterGen crew"""
        return Crew(
            agents=self.agents,
            tasks=self.tasks,
            process=Process.sequential,
            verbose=2,
        )

#!/usr/bin/env python
from newsletter_gen.crew import NewsletterGenCrew
import os

def load_html_template():
    with open("src/newsletter_gen/config/newsletter_template.html", "r") as file:
        html_template = file.read()
    return html_template


def run():  
    inputs = {
        "topic": input("Enter the topic: "),
        "html_template": load_html_template(),
    }
    NewsletterGenCrew().crew().kickoff(inputs=inputs)

$ poetry lock
$ poetry install
$ poetry run <YOUR_CREW_NAME>