From b273f50ac62a8d308f6b42e051cfd88863c72f38 Mon Sep 17 00:00:00 2001 From: qinyiwei Date: Thu, 29 Dec 2022 17:58:59 -0500 Subject: [PATCH 1/4] add react --- softwares/react/prompts/prompts_naive.json | 1 + softwares/react/react.py | 367 +++++++++++++++++++++ softwares/react/react_test.py | 26 ++ softwares/react/software_info.json | 54 +++ 4 files changed, 448 insertions(+) create mode 100644 softwares/react/prompts/prompts_naive.json create mode 100644 softwares/react/react.py create mode 100644 softwares/react/react_test.py create mode 100644 softwares/react/software_info.json diff --git a/softwares/react/prompts/prompts_naive.json b/softwares/react/prompts/prompts_naive.json new file mode 100644 index 0000000..df90974 --- /dev/null +++ b/softwares/react/prompts/prompts_naive.json @@ -0,0 +1 @@ +{"webthink_simple": "\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening. Milhouse is Bart Simpson's best friend in Mrs. Krabappel's fourth grade class at Springfield Elementary School. He is an insecure, gullible, and less popular child than Bart who is often led into trouble by Bart, who takes advantage of his friend's na\u00c3\u00afvet\u00c3\u00a9. Milhouse is a regular target for school bully Nelson Muntz and his friends Jimbo Jones, Dolph Starbeam and Kearney Zzyzwicz. Milhouse has a crush on Bart's sister, Lisa, a common plot element.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was designed by Matt Groening for a planned series on NBC, which was abandoned.[4] The design was then used for a Butterfinger commercial, and it was decided to use the character in the series.[5][6] Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. The name was the most \"unfortunate name Matt Groening could think of for a kid\".[1] Years earlier, in a 1986 Life in Hell comic entitled \"What to Name the Baby\", Groening listed Milhouse as a name \"no longer recommended\".[7] Milhouse is a favorite among the staff as Al Jean noted \"most of the writers are more like Milhouse than Bart\".[1] His last name was given to him by Howard Gewirtz, a freelance writer who wrote the episode \"Homer Defined\". Gewirtz got the name from one of his wife's friends.[8] According to Arden Myrin and Dana Gould (former writer and co-executive producer of The Simpsons), Rob Cohen (Simpsons writer) inspired Milhouse's look.[2]\nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?\nThought 1: I need to search Guitars for Wounded Warriors, then find in which New York county it was recorded.\nAction 1: Search[Guitars for Wounded Warriors]\nObservation 1: Guitars for Wounded Warriors is the sixth solo studio album by guitarist Brian Tarquin, released in August 2014 by Cleopatra Records. In 2016, it received a Best Compilation Album nomination by the Independent Music Awards. All tracks were recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. Being moved by the lack of support for returning veterans through his life Tarquin decided to compose a heavy instrumental rock album as a way to show his appreciation to all veterans. So once again he enlisted top notch players to guest with him on the album, Billy Sheehan on Black Hawk, Ron \"Bumblefoot\" Thal and Reb Beach on Taliban Terror, Steve Morse on Freedom, Chris Poland on Charlie Surfs and Hunting, Chuck Loeb on Escape Kabul, Hal Lindes on Sand & Blood, Gary Hoey on 5 Klicks To Hell and Baghdad, Alex De Rosso Dokken on Broken Arrow, and The Flyin' Ryan Brothers on Surgical Strike. The entire album was engineered, produced and composed by Tarquin especially for each one of the guest guitarists. Partial proceeds are donated to the Fisher House Foundation from sales.\nThought 2: Guitars for Wounded Warriors was recorded in New Paltz (village), New York. I need to search New Paltz (village), New York and find which county it is in.\nAction 2: Search[New Paltz (village), New York]\nObservation 2: New Paltz is a village in Ulster County located in the U.S. state of New York. It is approximately 80 miles (130 km) north of New York City and 70 miles (110 km) south of Albany. The population was 7,324 at the 2020 census.[3]\nThought 3: New Paltz (village), New York is in Ulster County, so the answer is Ulster County.\nAction 3: Finish[Ulster County]\nQuestion: This American actor who starred in \"Veep\" was also in an action comedy film directed by Seth Rogen and Evan Goldberg heavily inspired by what?\nThought 1: There could be many American actors who starred in \"Veep\". So I can first search \"directed by Seth Rogen and Evan Goldberg\" and find action comedy films directed by Seth Rogen and Evan Goldberg.\nAction 1: Search[directed by Seth Rogen and Evan Goldberg]\nObservation 1: Could not find directed by Seth Rogen and Evan Goldberg. Similar: ['The Interview (redirect from The Interview (Seth Rogen film))', 'Evan Goldberg', 'Jay and Seth Versus the Apocalypse', 'This Is the End (category Films directed by Seth Rogen)', 'Seth Rogen filmography', 'Superbad (category Films with screenplays by Seth Rogen)', 'Pineapple Express (film) (category Films with screenplays by Seth Rogen)', 'Sausage Party (category Films produced by Seth Rogen)', 'Untitled Teenage Mutant Ninja Turtles film (category Films produced by Seth Rogen)', 'The Green Hornet (2011 film) (category Films produced by Seth Rogen)', 'Good Boys (film) (category Films produced by Seth Rogen)', 'Seth Rogen', 'Neighbors 2: Sorority Rising (category Films produced by Seth Rogen)', 'An American Pickle (category Films produced by Seth Rogen)', 'The Watch (2012 film) (category Films with screenplays by Seth Rogen)', 'Neighbors (2014 film) (category Films produced by Seth Rogen)', 'The Night Before (2015 film) (category Films produced by Seth Rogen)', 'Martin Starr', 'Preacher (TV series) (category Television shows directed by Steph Green)', 'Long Shot (2019 film) (category Films produced by Seth Rogen)'].\nThought 2: I can check search The Interview (Seth Rogen film), Jay and Seth Versus the Apocalypse, This Is the End (category Films directed by Seth Rogen) and find which is an action comedy film directed by Seth Rogen and Evan Goldberg.\nAction 2: Search[The Interview (Seth Rogen film)]\nObservation 2: The Interview is a 2014 action-adventure comedy film co-produced and directed by Seth Rogen and Evan Goldberg in their second directorial work, following This Is the End (2013). The screenplay was written by Dan Sterling, based on a story he co-wrote with Rogen and Goldberg. The film stars Rogen and James Franco as journalists who set up an interview with North Korean leader Kim Jong-un (Randall Park), and are then recruited by the CIA to assassinate him. The film is inspired by a 2012 Vice documentary.\nThought 3: The Interview is an action comedy film directed by Seth Rogen and Evan Goldberg, and it is inspired by a 2012 Vice documentary. So I can probably stop now without checking other films, and the answer is probably a 2012 Vice documentary.\nAction 3: Finish[a 2012 Vice documentary]\nQuestion: The head coach during the 1997\u201398 Indiana Pacers season retired as a player from what NBA team?\nThought 1: I can search Indiana Pacers, find the head coach during the 1997\u201398 season, then find from what NBA team it is retired as a player.\nAction 1: Search[Indiana Pacers]\nObservation 1: The Indiana Pacers are an American professional basketball team based in Indianapolis. The Pacers compete in the National Basketball Association (NBA) as a member of the league's Eastern Conference Central Division. The Pacers were first established in 1967 as a member of the American Basketball Association (ABA) and became a member of the NBA in 1976 as a result of the ABA\u00e2\u0080\u0093NBA merger. They play their home games at Gainbridge Fieldhouse. The team is named after the state of Indiana's history with the Indianapolis 500's pace cars and with the harness racing industry.[8][9]\nThought 2: I can look up \"1997\u201398\" to find the head coach during the 1997\u201398 season.\nAction 2: Lookup[1997\u201398]\nObservation 2: No more results.\nThought 3: I can look up \"1997\" instead to find the head coach during the 1997\u201398 season\nAction 3: Lookup[1997]\nObservation 3: (Result 1 / 5) In the 1997\u00e2\u0080\u009398 NBA season, Indiana native and former Boston Celtics great Larry Bird was hired as head coach. He led the Pacers to a 19-game improvement over the previous season, finishing 58\u00e2\u0080\u009324 \u00e2\u0080\u0093 at the time, the most the franchise had ever won as an NBA team, and tying the 1970\u00e2\u0080\u009371 ABA Pacers for the franchise record. Chris Mullin joined the team in the off-season and immediately became a valuable part of the Pacers lineup\u00e2\u0080\u0094and starting small forward. Assistant coaches Rick Carlisle, in charge of the offense, and Dick Harter, who coached the defense, were key in getting the most out of the Pacers' role players such as Dale Davis, Derrick McKey and a young Antonio Davis. Miller and Rik Smits both made the All-Star team that year, and in the playoffs, the Pacers breezed past the Cleveland Cavaliers and New York Knicks before falling to the Chicago Bulls in a seven-game Eastern Conference Finals.\nThought 4: In the 1997-98 season, former Boston Celtics great Larry Bird was hired as head coach. So the answer is Boston Celtics.\nAction 4: Finish[Boston Celtics]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause. He is appreciated for many narrative features produced between 1947 and 1963 including They Live By Night, In A Lonely Place, Johnny Guitar, and Bigger Than Life, as well as an experimental work produced throughout the 1970s titled We Can't Go Home Again, which was unfinished at the time of Ray's death.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan (/\u02c8i\u02d0li\u0259 k\u0259\u02c8z\u00e6n/;[1][2] born Elias Kazantzoglou (Greek: \u0397\u03bb\u03af\u03b1\u03c2 \u039a\u03b1\u03b6\u03b1\u03bd\u03c4\u03b6\u03cc\u03b3\u03bb\u03bf\u03c5);[3] September 7, 1909 \u2013 September 28, 2003) was an American film and theatre director, producer, screenwriter and actor, described by The New York Times as \"one of the most honored and influential directors in Broadway and Hollywood history\".[4]\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]\nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey.[2] In 2011 the circulation of the magazine was 1,310,696 copies.[3]\nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (\u00d0\u009f\u00d0\u00b0\u00cc\u0081\u00d0\u00b2\u00d0\u00b5\u00d0\u00bb \u00d0\u00a1\u00d0\u00b0\u00d0\u00bc\u00d1\u0083\u00d0\u00b8\u00cc\u0081\u00d0\u00bb\u00d0\u00be\u00d0\u00b2\u00d0\u00b8\u00d1\u0087 \u00d0\u00a3\u00d1\u0080\u00d1\u008b\u00d1\u0081\u00d0\u00be\u00cc\u0081\u00d0\u00bd) (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory, and for developing Urysohn's metrization theorem and Urysohn's lemma, both of which are fundamental results in topology. His name is also commemorated in the terms Urysohn universal space, Fr\u00c3\u00a9chet\u00e2\u0080\u0093Urysohn space, Menger\u00e2\u0080\u0093Urysohn dimension and Urysohn integral equation. He and Pavel Alexandrov formulated the modern definition of compactness in 1923.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin (/le\u00c9\u00aa.o\u00ca\u008a\u00cb\u0088ni\u00cb\u0090d \u00cb\u0088l\u00c9\u009bv\u00c9\u00aan/ lay-oh-NEED LEV-in; Russian: \u00d0\u009b\u00d0\u00b5\u00d0\u00be\u00d0\u00bd\u00d0\u00b8\u00cc\u0081\u00d0\u00b4 \u00d0\u0090\u00d0\u00bd\u00d0\u00b0\u00d1\u0082\u00d0\u00be\u00cc\u0081\u00d0\u00bb\u00d1\u008c\u00d0\u00b5\u00d0\u00b2\u00d0\u00b8\u00d1\u0087 \u00d0\u009b\u00d0\u00b5\u00cc\u0081\u00d0\u00b2\u00d0\u00b8\u00d0\u00bd; Ukrainian: \u00d0\u009b\u00d0\u00b5\u00d0\u00be\u00d0\u00bd\u00d1\u0096\u00cc\u0081\u00d0\u00b4 \u00d0\u0090\u00d0\u00bd\u00d0\u00b0\u00d1\u0082\u00d0\u00be\u00cc\u0081\u00d0\u00bb\u00d1\u0096\u00d0\u00b9\u00d0\u00be\u00d0\u00b2\u00d0\u00b8\u00d1\u0087 \u00d0\u009b\u00d0\u00b5\u00cc\u0081\u00d0\u00b2\u00d1\u0096\u00d0\u00bd; born November 2, 1948) is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell Jr.', 'Adam Clayton Powell Sr.', 'Adam Clayton Powell IV', 'Adam Clayton Powell III', 'Seventh Avenue (Manhattan) (redirect from Adam Clayton Powell Jr. Boulevard)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito', 'Harlem', 'St. Nicholas Historic District', 'Abyssinian Baptist Church', 'Powell (surname)', 'Adam (musical)', 'Statue of Adam Clayton Powell Jr.', 'List of people from Harlem', 'Adam (given name)', 'Five-Percent Nation', 'List of American liberals'].\nThought 2: I can search Adam Clayton Powell (film) to find the documentary.\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\n", "cotqa_simple": "\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought: Let's think step by step. Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAnswer: Richard Nixon\nQuestion: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?\nThought: Let's think step by step. Guitars for Wounded Warriors was recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. New Paltz is a village in Ulster County located in the U.S. state of New York. So the answer is Ulster County.\nAnswer: Ulster County\nQuestion: This American actor who starred in \"Veep\" was also in an action comedy film directed by Seth Rogen and Evan Goldberg heavily inspired by what?\nThought: Let's think step by step. The Interview is an action comedy film directed by Seth Rogen and Evan Goldberg. It is inspired by a 2012 Vice documentary. So the answer is a 2012 Vice documentary.\nAnswer: a 2012 Vice documentary\nQuestion: The head coach during the 1997\u201398 Indiana Pacers season retired as a player from what NBA team?\nThought: Let's think step by step. The head coach during the 1997\u201398 Indiana Pacers season was Larry Bird. Larry Bird retired as a player from Boston Celtics. So the answer is Boston Celtics.\nAnswer: Boston Celtics\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought: Let's think step by step. Professions of Nicholas Ray are director, screenwriter, and actor. Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAnswer: director, screenwriter, actor\nQuestion: Which band was founded first, Hole, the rock band that Courtney Love was a frontwoman of, or The Wolfhounds?\nThought: Let's think step by step. Hole was founded in 1989. The Wolfhounds was founded in 1985. 1989 (Hole) > 1985 (The Wolfhounds), so The Wolfhounds was founded first.\nAnswer: The Wolfhounds\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought: Let's think step by step. Pavel Urysohn is a mathematician. Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work.\nAnswer: Yes\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought: Let's think step by step. Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAnswer: The Saimaa Gesture\n", "webqa_simple": "\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nAnswer: Richard Nixon\nQuestion: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?\nAnswer: Ulster County\nQuestion: This American actor who starred in \"Veep\" was also in an action comedy film directed by Seth Rogen and Evan Goldberg heavily inspired by what?\nAnswer: 2012 Vice documentary\nQuestion: The head coach during the 1997\u201398 Indiana Pacers season retired as a player from what NBA team?\nAnswer: Boston Celtics\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nAnswer: director, screenwriter, actor\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nAnswer: Arthur's Magazine\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nAnswer: yes\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nAnswer: The Saimaa Gesture\n", "webthink_simple_3": "\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas. This took place from 1780 to 1650 million years ago (Mya), during the Paleoproterozoic (Statherian Period). It is recorded in the Colorado orogen, a >500-km-wide belt of oceanic arc rock that extends southward into New Mexico. The Colorado orogeny was likely part of the larger Yavapai orogeny.\nThought 2: It does not mention the eastern sector of the Colorado orogeny. I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The Colorado orogen, formerly called the Colorado province, is a >500-km-wide belt of oceanic arc rock (1.78\u20131.65 Ga) that extends southward into New Mexico and composes a major part of the Proterozoic provinces of southwestern United States. This transcontinental collisional event occurred during the Paleoproterozoic (Statherian Period).[1] The Wyoming sector of the Colorado orogeny was formerly called the Medicine Bow orogeny. The eastern sector extends into the High Plains and is called the Central Plains orogeny. The boundary between the Colorado orogeny and the Wyoming craton is the Cheyenne belt, a 5-km-wide mylonitic shear zone that verges northward. The Cheyenne belt transects and cuts off the south edge of the older Trans-Hudson orogeny.[2]\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains, mainly in the Western United States, but also partly in the Midwest states of Nebraska, Kansas, and South Dakota, generally encompassing the western part of the Great Plains before the region reaches the Rocky Mountains. The High Plains are located in eastern Montana, southeastern Wyoming, southwestern South Dakota, western Nebraska, eastern Colorado, western Kansas, eastern New Mexico, western Oklahoma, and to just south of the Texas Panhandle.[4] The southern region of the Western High Plains ecology region contains the geological formation known as Llano Estacado which can be seen from a short distance or on satellite maps.[5] From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?\nThought 1: I need to search Guitars for Wounded Warriors, then find in which New York county it was recorded.\nAction 1: Search[Guitars for Wounded Warriors]\nObservation 1: Guitars for Wounded Warriors is the sixth solo studio album by guitarist Brian Tarquin, released in August 2014 by Cleopatra Records. In 2016, it received a Best Compilation Album nomination by the Independent Music Awards. All tracks were recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. Being moved by the lack of support for returning veterans through his life Tarquin decided to compose a heavy instrumental rock album as a way to show his appreciation to all veterans. So once again he enlisted top notch players to guest with him on the album, Billy Sheehan on Black Hawk, Ron \"Bumblefoot\" Thal and Reb Beach on Taliban Terror, Steve Morse on Freedom, Chris Poland on Charlie Surfs and Hunting, Chuck Loeb on Escape Kabul, Hal Lindes on Sand & Blood, Gary Hoey on 5 Klicks To Hell and Baghdad, Alex De Rosso Dokken on Broken Arrow, and The Flyin' Ryan Brothers on Surgical Strike. The entire album was engineered, produced and composed by Tarquin especially for each one of the guest guitarists. Partial proceeds are donated to the Fisher House Foundation from sales.\nThought 2: Guitars for Wounded Warriors was recorded in New Paltz (village), New York. I need to search New Paltz (village), New York and find which county it is in.\nAction 2: Search[New Paltz (village), New York]\nObservation 2: New Paltz is a village in Ulster County located in the U.S. state of New York. It is approximately 80 miles (130 km) north of New York City and 70 miles (110 km) south of Albany. The population was 7,324 at the 2020 census.[3]\nThought 3: New Paltz (village), New York is in Ulster County, so the answer is Ulster County.\nAction 3: Finish[Ulster County]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]\nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey.[2] In 2011 the circulation of the magazine was 1,310,696 copies.[3]\nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\n", "webthink_simple6": "\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n", "webact_simple6": "Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nAction 3: Finish[yes]\n\n", "cotqa_simple6": "Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought: Let's think step by step. The eastern sector of Colorado orogeny extends into the High Plains. High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAnswer: 1,800 to 7,000 ft\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought: Let's think step by step. Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAnswer: Richard Nixon\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought: Let's think step by step. Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAnswer: The Saimaa Gesture\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought: Let's think step by step. Professions of Nicholas Ray are director, screenwriter, and actor. Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAnswer: director, screenwriter, actor\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought: Let's think step by step. Arthur's Magazine was started in 1844. First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAnswer: Arthur's Magazine\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought: Let's think step by step. Pavel Urysohn is a mathematician. Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work.\nAnswer: Yes\n", "webqa_simple6": "Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nAnswer: 1,800 to 7,000 ft\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nAnswer: Richard Nixon\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nAnswer: The Saimaa Gesture\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nAnswer: director, screenwriter, actor\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nAnswer: Arthur's Magazine\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nAnswer: Yes\n"} \ No newline at end of file diff --git a/softwares/react/react.py b/softwares/react/react.py new file mode 100644 index 0000000..5eac9d0 --- /dev/null +++ b/softwares/react/react.py @@ -0,0 +1,367 @@ +from __future__ import annotations + +from promptware.constants.tasks import TaskType +from promptware.info import SoftwareInfo +from promptware.kernels.plm import PLMKernelConfig +from promptware.licenses import LicenseType +from promptware.promptware import PromptConfig, Promptware +import json + +import numpy as np +import os +import re +import string +import json +import time +import gym +import requests +from bs4 import BeautifulSoup + + +INSTRUCTION = """Solve a question answering task with interleaving Thought, Action, +Observation steps. Thought can reason about the current situation, and Action can be three types: +(1) Search[entity], which searches the exact entity on Wikipedia and returns the first +paragraph if it exists. If not, it will return some similar entities to search. +(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage. +(3) Finish[answer], which returns the answer and finishes the task. +Here are some examples. +""" + +folder = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) + '/prompts/' +prompt_file = 'prompts_naive.json' +with open(folder + prompt_file, 'r') as f: + PROMPT_DICT = json.load(f) + + +def step(env, action): + attempts = 0 + while attempts < 10: + try: + return env.step(action) + except requests.exceptions.Timeout: + attempts += 1 + + +class ReactPromptware(Promptware): + def _info(self) -> SoftwareInfo: + return SoftwareInfo( + description="This promptware is used to answer a question" + " that involves multiple step reasoning with" + " chain-of-thought method.", + creator="Promptware Authors", + homepage="https://github.com/expressai/promptware", + reference="", + codebase_url="https://github.com/expressai/promptware/tree/main/softwares", + license=LicenseType.apache_2_0, + task=TaskType.qa_open_domain, + ) + + def _kernel_configs(self): + return { + "openai": PLMKernelConfig( + platform="openai", + model_name="text-davinci-002", + temperature=0, + max_tokens=100, + top_p=1, + frequency_penalty=0.0, + presence_penalty=0.0, + ) + } + + def _software_configs(self): + return { + "multi_hop_qa": PromptConfig( + name="multi_hop_qa", + description="This promptware is used to answer a question" + " that involves multiple step reasoning with" + " react method.", + instruction=INSTRUCTION, + demonstration=[PROMPT_DICT['webthink_simple6']], + prompt_template=lambda input: f"Question: {input['question']}", + task=TaskType.qa_open_domain, + ) + } + + def execute(self, input): + env = WikiEnv() + env = HotPotQAWrapper(env) + env = LoggingWrapper(env) + + kernel = self.kernel_configs["openai"].to_kernel() + # code += question + "\n" + code = self.get_code(input, self.default_software_config) + env.set(question=input['question']) + n_calls, n_badcalls = 0, 0 + for i in range(1, 8): + n_calls += 1 + kernel.config.stop = [f"\nObservation {i}:"] + thought_action = kernel.execute( + code + f"Thought {i}:") + thought_action = self.normalize_output(thought_action["text"]) + try: + thought, action = thought_action.strip().split(f"\nAction {i}: ") + except: + print('ohh...', thought_action) + n_badcalls += 1 + n_calls += 1 + thought = thought_action.strip().split('\n')[0] + kernel.config['stop'] = [f"\n"] + action = kernel.execute( + code + f"Thought {i}: {thought}\nAction {i}:") + action = self.normalize_output(action["text"]) + obs, r, done, info = step(env, action[0].lower() + action[1:]) + obs = obs.replace('\\n', '') + step_str = f"Thought {i}: {thought}\nAction {i}: {action}\nObservation {i}: {obs}\n" + code += step_str + + print(step_str) + if done: + break + if not done: + obs, r, done, info = step(env, "finish[]") + + print(info, '\n') + info.update({'n_calls': n_calls, 'n_badcalls': n_badcalls, 'traj': code}) + return info['answer'] + + def _example(self): + return { + "input": { + "question": "What movie did actress Irene Jacob " + "complete before the American action crime thriller " + "film directed by Stuart Bird?" + }, + "output": "Beyond the Clouds", + } + + +def normalize_answer(s): + def remove_articles(text): + return re.sub(r"\b(a|an|the)\b", " ", text) + + def white_space_fix(text): + return " ".join(text.split()) + + def remove_punc(text): + exclude = set(string.punctuation) + return "".join(ch for ch in text if ch not in exclude) + + def lower(text): + return text.lower() + + return white_space_fix(remove_articles(remove_punc(lower(s)))) + + +class HotPotQAWrapper(gym.Wrapper): + def __init__(self, env): + super().__init__(env) + + def set(self, question, answer=None): + self.data = (question, answer) + + def _get_info(self): + return { + "steps": self.steps, + "answer": self.answer, + "question": self.data[0], + } + + def get_reward(self, info): + if info['answer'] is not None and self.data[1] is not None: + pred = normalize_answer(self.data[1]) + gt = normalize_answer(info['answer']) + score = (pred == gt) + return int(score) + return 0 + + def step(self, action): + # TODO: first step obs does not have question. + obs, _, done, info = self.env.step(action) + reward = self.get_reward(info) + if done: + obs = f"Episode finished, reward = {reward}\n" + info.update( + {"gt_answer": self.data[1]}) + return obs, reward, done, info + + def __len__(self): + return len(self.data) + + +def clean_str(p): + return p.encode().decode("unicode-escape").encode("latin1").decode("utf-8") + + +class textSpace(gym.spaces.Space): + def contains(self, x) -> bool: + """Return boolean specifying if x is a valid member of this space.""" + return isinstance(x, str) + + +class WikiEnv(gym.Env): + + def __init__(self): + """ + Initialize the environment. + """ + super().__init__() + self.page = None # current Wikipedia page + self.obs = None # current observation + self.lookup_keyword = None # current lookup keyword + self.lookup_list = None # list of paragraphs containing current lookup keyword + self.lookup_cnt = None # current lookup index + self.steps = 0 # current number of steps + self.answer = None # current answer from the agent + self.observation_space = self.action_space = textSpace() + self.search_time = 0 + self.num_searches = 0 + + def _get_obs(self): + return self.obs + + def _get_info(self): + return {"steps": self.steps, "answer": self.answer} + + def reset(self, seed=None, return_info=False, options=None): + # We need the following line to seed self.np_random + # super().reset(seed=seed) + self.obs = ("Interact with Wikipedia using search[], lookup[], and " + "finish[].\n") + self.page = None + self.lookup_keyword = None + self.lookup_list = None + self.lookup_cnt = None + self.steps = 0 + self.answer = None + observation = self._get_obs() + info = self._get_info() + return (observation, info) if return_info else observation + + def construct_lookup_list(self, keyword): + # find all paragraphs + if self.page is None: + return [] + paragraphs = self.page.split("\n") + paragraphs = [p.strip() for p in paragraphs if p.strip()] + + # find all sentence + sentences = [] + for p in paragraphs: + sentences += p.split('. ') + sentences = [s.strip() + '.' for s in sentences if s.strip()] + + parts = sentences + parts = [p for p in parts if keyword.lower() in p.lower()] + return parts + + @staticmethod + def get_page_obs(page): + # find all paragraphs + paragraphs = page.split("\n") + paragraphs = [p.strip() for p in paragraphs if p.strip()] + + # find all sentence + sentences = [] + for p in paragraphs: + sentences += p.split('. ') + sentences = [s.strip() + '.' for s in sentences if s.strip()] + return ' '.join(sentences[:5]) + + # ps = page.split("\n") + # ret = ps[0] + # for i in range(1, len(ps)): + # if len((ret + ps[i]).split(" ")) <= 50: + # ret += ps[i] + # else: + # break + # return ret + + def search_step(self, entity): + entity_ = entity.replace(" ", "+") + search_url = f"https://en.wikipedia.org/w/index.php?search={entity_}" + old_time = time.time() + response_text = requests.get(search_url).text + self.search_time += time.time() - old_time + self.num_searches += 1 + soup = BeautifulSoup(response_text, features="html.parser") + result_divs = soup.find_all("div", {"class": "mw-search-result-heading"}) + if result_divs: # mismatch + self.result_titles = [clean_str(div.get_text().strip()) + for div in result_divs] + self.obs = f"Could not find {entity}. Similar: {self.result_titles[:5]}." + else: + page = [p.get_text().strip() + for p in soup.find_all("p") + soup.find_all("ul")] + if any("may refer to:" in p for p in page): + self.search_step("[" + entity + "]") + else: + self.page = "" + for p in page: + if len(p.split(" ")) > 2: + self.page += clean_str(p) + if not p.endswith("\n"): + self.page += "\n" + self.obs = self.get_page_obs(self.page) + self.lookup_keyword = self.lookup_list = self.lookup_cnt = None + + def step(self, action): + reward = 0 + done = False + action = action.strip() + if self.answer is not None: # already finished + done = True + return self.obs, reward, done, self._get_info() + + if action.startswith("search[") and action.endswith("]"): + entity = action[len("search["):-1] + # entity_ = entity.replace(" ", "_") + # search_url = f"https://en.wikipedia.org/wiki/{entity_}" + self.search_step(entity) + elif action.startswith("lookup[") and action.endswith("]"): + keyword = action[len("lookup["):-1] + if self.lookup_keyword != keyword: # reset lookup + self.lookup_keyword = keyword + self.lookup_list = self.construct_lookup_list(keyword) + self.lookup_cnt = 0 + if self.lookup_cnt >= len(self.lookup_list): + self.obs = "No more results.\n" + else: + self.obs = f"(Result {self.lookup_cnt + 1} / {len(self.lookup_list)}) " + \ + self.lookup_list[self.lookup_cnt] + self.lookup_cnt += 1 + elif action.startswith("finish[") and action.endswith("]"): + answer = action[len("finish["):-1] + self.answer = answer + done = True + self.obs = f"Episode finished, reward = {reward}\n" + elif action.startswith("think[") and action.endswith("]"): + self.obs = "Nice thought." + else: + self.obs = "Invalid action: {}".format(action) + + self.steps += 1 + + return self.obs, reward, done, self._get_info() + + def get_time_info(self): + speed = self.search_time / self.num_searches if self.num_searches else 0 + return { + "call_speed": speed, + "call_time": self.search_time, + "num_calls": self.num_searches, + } + + +class LoggingWrapper(gym.Wrapper): + def __init__(self, env, folder="trajs", file_id=None): + super().__init__(env) + self.trajs = [] + self.traj = {"observations": [], "actions": []} + self.folder = folder + self.file_id = np.random.randint(0, 10000000) if file_id is None else file_id + self.file_path = f"{self.folder}/{self.file_id}.json" + os.makedirs("trajs", exist_ok=True) + + def __len__(self): + return len(self.env.data) diff --git a/softwares/react/react_test.py b/softwares/react/react_test.py new file mode 100644 index 0000000..4afdd81 --- /dev/null +++ b/softwares/react/react_test.py @@ -0,0 +1,26 @@ +import os +import sys +import unittest + +current_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) +sys.path.append(current_path) + +from react import ReactPromptware # noqa + + +class TestReactPromptware(unittest.TestCase): + def test_write_to_directory(self): + software = ReactPromptware() + file_path = software.info.write_to_directory(current_path) + self.assertTrue(os.path.exists(file_path)) + + def test_execute_default(self): + software = ReactPromptware(config_name="default") + input = { + "question": "What movie did actress Irene Jacob " + "complete before the American action crime thriller " + "film directed by Stuart Bird?" + } + result = software.execute(input) + print(result) + self.assertGreater(len(result), 0) diff --git a/softwares/react/software_info.json b/softwares/react/software_info.json new file mode 100644 index 0000000..9545e28 --- /dev/null +++ b/softwares/react/software_info.json @@ -0,0 +1,54 @@ +{ + "description": "This promptware is used to answer a question that involves multiple step reasoning with chain-of-thought method.", + "creator": "Promptware Authors", + "homepage": "https://github.com/expressai/promptware", + "reference": "", + "codebase_url": "https://github.com/expressai/promptware/tree/main/softwares", + "license": "apache-2.0", + "task": "qa-open-domain", + "research_tasks": null, + "application_categories": null, + "application_subcategories": null, + "original_platform": null, + "design_pattern": null, + "source_language": null, + "target_language": null, + "module_name": "react_promptware", + "config_name": "default", + "kernel_configs": { + "openai": { + "platform": "openai", + "model_name": "text-davinci-002", + "max_tokens": 100, + "temperature": 0, + "top_p": 1, + "suffix": "", + "frequency_penalty": 0.0, + "presence_penalty": 0.0, + "logprobs": 1, + "n": 1, + "echo": false, + "stop": null + } + }, + "software_configs": { + "multi_hop_qa": { + "name": "multi_hop_qa", + "description": "This promptware is used to answer a question that involves multiple step reasoning with react method.", + "instruction": "Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n", + "demonstration": [ + "\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n" + ], + "prompt_template": "lambda input: f\"Question: {input['question']}\",", + "task": "qa-open-domain" + } + }, + "dataset_configs": null, + "example": { + "input": { + "question": "What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?" + }, + "output": "Beyond the Clouds" + }, + "example_prompt": "Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n\n\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n\nQuestion: What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?\n" +} \ No newline at end of file From afd96feb24dfc556097c1556c0c97e7adddec1c7 Mon Sep 17 00:00:00 2001 From: qinyiwei Date: Thu, 12 Jan 2023 23:36:35 -0500 Subject: [PATCH 2/4] add auto_cot --- softwares/auto_cot/auto_cot.py | 83 ++++++++++++++++++++++++ softwares/auto_cot/auto_cot_test.py | 27 ++++++++ softwares/auto_cot/demos/multiarith_auto | 52 +++++++++++++++ softwares/auto_cot/software_info.json | 47 ++++++++++++++ 4 files changed, 209 insertions(+) create mode 100644 softwares/auto_cot/auto_cot.py create mode 100644 softwares/auto_cot/auto_cot_test.py create mode 100644 softwares/auto_cot/demos/multiarith_auto create mode 100644 softwares/auto_cot/software_info.json diff --git a/softwares/auto_cot/auto_cot.py b/softwares/auto_cot/auto_cot.py new file mode 100644 index 0000000..dcb5a7c --- /dev/null +++ b/softwares/auto_cot/auto_cot.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +from promptware.info import SoftwareInfo +from promptware.kernels.plm import PLMKernelConfig +from promptware.licenses import LicenseType +from promptware.promptware import PromptConfig, Promptware +from promptware.tasks import TaskType + +import json +import os + +folder = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) + '/demos/' + + +def create_demo_text(demo_path): + x, z, y = [], [], [] + + with open(demo_path, encoding="utf-8") as f: + json_data = json.load(f) + json_data = json_data["demo"] + for line in json_data: + x.append(line["question"]) + z.append(line["rationale"]) + y.append(line["pred_ans"]) + + index_list = list(range(len(x))) + + demo_text = "" + for i in index_list: + demo_text += x[i] + " " + z[i] + " " + \ + "The answer is" + " " + y[i] + ".\n\n" + + return demo_text + + +class AutoCotPromptware(Promptware): + def _info(self) -> SoftwareInfo: + return SoftwareInfo( + description="This promptware is used to answer a question" + " that involves reasoning with chain-of-thought method.", + creator="Promptware Authors", + homepage="https://github.com/expressai/promptware", + reference="https://github.com/amazon-science/auto-cot", + codebase_url="https://github.com/expressai/promptware/tree/main/softwares", + license=LicenseType.apache_2_0, + task=TaskType.qa_open_domain, + ) + + def _kernel_configs(self): + return { + "openai": PLMKernelConfig( + platform="openai", + model_name="text-davinci-002", + temperature=0, + max_tokens=256, + top_p=1, + frequency_penalty=0.0, + presence_penalty=0.0, + ) + } + + def _software_configs(self): + return { + "multiarith": PromptConfig( + name="multiarith", + description="This promptware is used to answer an arithmatic question" + " that involves reasoning with Auto-CoT method.", + instruction="", + demonstration=[create_demo_text(folder+"multiarith_auto")], + prompt_template=lambda input: f"Q: {input['question']}\nA: Let's think step by step.", + task=TaskType.qa_open_domain, + ) + } + + def _example(self): + return { + "input": { + "question": "In a video game, each enemy defeated gives you 7 points. " + "If a level has 11 enemies total and you destroy all but 8 of them, " + "how many points would you earn?" + }, + "output": 21.0, + } diff --git a/softwares/auto_cot/auto_cot_test.py b/softwares/auto_cot/auto_cot_test.py new file mode 100644 index 0000000..3338a5c --- /dev/null +++ b/softwares/auto_cot/auto_cot_test.py @@ -0,0 +1,27 @@ +import os +import sys +import unittest + +current_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) +sys.path.append(current_path) + +from auto_cot import AutoCotPromptware # noqa + + +class TestAutoCotPromptware(unittest.TestCase): + def test_write_to_directory(self): + software = AutoCotPromptware() + file_path = software.info.write_to_directory(current_path) + self.assertTrue(os.path.exists(file_path)) + ''' + def test_execute_default(self): + software = AutoCotPromptware(config_name="default") + input = { + "question": "In a video game, each enemy defeated gives you 7 points. " + "If a level has 11 enemies total and you destroy all but 8 of them, " + "how many points would you earn?" + } + result = software.execute(input) + print(result) + self.assertGreater(len(result), 0) + ''' diff --git a/softwares/auto_cot/demos/multiarith_auto b/softwares/auto_cot/demos/multiarith_auto new file mode 100644 index 0000000..3367092 --- /dev/null +++ b/softwares/auto_cot/demos/multiarith_auto @@ -0,0 +1,52 @@ +{ + "demo": [ + { + "question": "Q: Wendy uploaded 45 pictures to Facebook. She put 27 pics into one album and put the rest into 9 different albums. How many pictures were in each album?\nA:", + "rationale": "Let's think step by step. First, we know that Wendy uploaded 45 pictures in total. Second, we know that Wendy put 27 pictures into one album. That means that Wendy put the remaining 18 pictures into 9 different albums. That means that each album would have 2 pictures.", + "pred_ans": "2", + "gold_ans": "2" + }, + { + "question": "Q: For Halloween Katie and her sister combined the candy they received. Katie had 8 pieces of candy while her sister had 23. If they ate 8 pieces the first night, how many pieces do they have left?\nA:", + "rationale": "Let's think step by step. Katie and her sister have a total of 8 + 23 = 31 pieces of candy. If they eat 8 pieces the first night, they have 31 - 8 = 23 pieces left.", + "pred_ans": "23", + "gold_ans": "23" + }, + { + "question": "Q: Bianca was organizing her book case making sure each of the shelves had exactly 8 books on it. If she had 5 shelves of mystery books and 4 shelves of picture books, how many books did she have total?\nA:", + "rationale": "Let's think step by step. There are 5 shelves of mystery books. Each shelf has 8 books. So that's 40 mystery books. There are 4 shelves of picture books. Each shelf has 8 books. So that's 32 picture books. 40 + 32 = 72 books.", + "pred_ans": "72", + "gold_ans": "72" + }, + { + "question": "Q: A pet store had 78 puppies. In one day they sold 30 of them and put the rest into cages with 8 in each cage. How many cages did they use?\nA:", + "rationale": "Let's think step by step. There are 78 puppies. 30 are sold, so that means there are 48 left. 48 divided by 8 is 6, so that means there are 6 cages with 8 puppies in each.", + "pred_ans": "6", + "gold_ans": "6" + }, + { + "question": "Q: A waiter had 14 customers to wait on. If 3 customers left and he got another 39 customers, how many customers would he have?\nA:", + "rationale": "Let's think step by step. The waiter had 14 customers to wait on. If 3 customers left, that means he would have 11 customers left. If he got another 39 customers, that means he would have 50 customers in total.", + "pred_ans": "50", + "gold_ans": "50" + }, + { + "question": "Q: A trivia team had 7 members total, but during a game 2 members didn't show up. If each member that did show up scored 4 points, how many points were scored total?\nA:", + "rationale": "Let's think step by step. There were 7 members on the team, but 2 members didn't show up. That means that there were 5 members that did show up. Each member that showed up scored 4 points. So if 5 members each scored 4 points, then the total number of points scored would be 5*4=20.", + "pred_ans": "20", + "gold_ans": "20" + }, + { + "question": "Q: Gwen had 18 math problems and 11 science problems for homework. If she finished 24 of the problems at school, how many problems did she have to do for homework?\nA:", + "rationale": "Let's think step by step. Gwen had 18 math problems and 11 science problems for homework. That means she had a total of 29 problems for homework. If she finished 24 of the problems at school, that means she had 5 problems left to do for homework.", + "pred_ans": "5", + "gold_ans": "5" + }, + { + "question": "Q: Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left?\nA:", + "rationale": "Let's think step by step. Mike made $69 from mowing lawns. He spent $24 on new mower blades. That means he has $45 left. Each game costs $5, so he could buy 9 games.", + "pred_ans": "9", + "gold_ans": "9" + } + ] +} \ No newline at end of file diff --git a/softwares/auto_cot/software_info.json b/softwares/auto_cot/software_info.json new file mode 100644 index 0000000..3e89df4 --- /dev/null +++ b/softwares/auto_cot/software_info.json @@ -0,0 +1,47 @@ +{ + "description": "This promptware is used to answer a question that involves reasoning with chain-of-thought method.", + "creator": "Promptware Authors", + "homepage": "https://github.com/expressai/promptware", + "reference": "https://github.com/amazon-science/auto-cot", + "codebase_url": "https://github.com/expressai/promptware/tree/main/softwares", + "license": "apache-2.0", + "task": "qa-open-domain", + "module_name": "auto_cot_promptware", + "config_name": "default", + "kernel_configs": { + "openai": { + "platform": "openai", + "model_name": "text-davinci-002", + "max_tokens": 256, + "temperature": 0, + "top_p": 1, + "suffix": "", + "frequency_penalty": 0.0, + "presence_penalty": 0.0, + "logprobs": 1, + "n": 1, + "echo": false, + "stop": null + } + }, + "software_configs": { + "multiarith": { + "name": "multiarith", + "description": "This promptware is used to answer an arithmatic question that involves reasoning with Auto-CoT method.", + "instruction": "", + "demonstration": [ + "Q: Wendy uploaded 45 pictures to Facebook. She put 27 pics into one album and put the rest into 9 different albums. How many pictures were in each album?\nA: Let's think step by step. First, we know that Wendy uploaded 45 pictures in total. Second, we know that Wendy put 27 pictures into one album. That means that Wendy put the remaining 18 pictures into 9 different albums. That means that each album would have 2 pictures. The answer is 2.\n\nQ: For Halloween Katie and her sister combined the candy they received. Katie had 8 pieces of candy while her sister had 23. If they ate 8 pieces the first night, how many pieces do they have left?\nA: Let's think step by step. Katie and her sister have a total of 8 + 23 = 31 pieces of candy. If they eat 8 pieces the first night, they have 31 - 8 = 23 pieces left. The answer is 23.\n\nQ: Bianca was organizing her book case making sure each of the shelves had exactly 8 books on it. If she had 5 shelves of mystery books and 4 shelves of picture books, how many books did she have total?\nA: Let's think step by step. There are 5 shelves of mystery books. Each shelf has 8 books. So that's 40 mystery books. There are 4 shelves of picture books. Each shelf has 8 books. So that's 32 picture books. 40 + 32 = 72 books. The answer is 72.\n\nQ: A pet store had 78 puppies. In one day they sold 30 of them and put the rest into cages with 8 in each cage. How many cages did they use?\nA: Let's think step by step. There are 78 puppies. 30 are sold, so that means there are 48 left. 48 divided by 8 is 6, so that means there are 6 cages with 8 puppies in each. The answer is 6.\n\nQ: A waiter had 14 customers to wait on. If 3 customers left and he got another 39 customers, how many customers would he have?\nA: Let's think step by step. The waiter had 14 customers to wait on. If 3 customers left, that means he would have 11 customers left. If he got another 39 customers, that means he would have 50 customers in total. The answer is 50.\n\nQ: A trivia team had 7 members total, but during a game 2 members didn't show up. If each member that did show up scored 4 points, how many points were scored total?\nA: Let's think step by step. There were 7 members on the team, but 2 members didn't show up. That means that there were 5 members that did show up. Each member that showed up scored 4 points. So if 5 members each scored 4 points, then the total number of points scored would be 5*4=20. The answer is 20.\n\nQ: Gwen had 18 math problems and 11 science problems for homework. If she finished 24 of the problems at school, how many problems did she have to do for homework?\nA: Let's think step by step. Gwen had 18 math problems and 11 science problems for homework. That means she had a total of 29 problems for homework. If she finished 24 of the problems at school, that means she had 5 problems left to do for homework. The answer is 5.\n\nQ: Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left?\nA: Let's think step by step. Mike made $69 from mowing lawns. He spent $24 on new mower blades. That means he has $45 left. Each game costs $5, so he could buy 9 games. The answer is 9.\n\n" + ], + "prompt_template": "lambda input: f\"Q: {input['question']}\\nA: Let's think step by step.\",", + "task": "qa-open-domain" + } + }, + "dataset_configs": null, + "example": { + "input": { + "question": "In a video game, each enemy defeated gives you 7 points. If a level has 11 enemies total and you destroy all but 8 of them, how many points would you earn?" + }, + "output": 21.0 + }, + "example_prompt": "Q: Wendy uploaded 45 pictures to Facebook. She put 27 pics into one album and put the rest into 9 different albums. How many pictures were in each album?\nA: Let's think step by step. First, we know that Wendy uploaded 45 pictures in total. Second, we know that Wendy put 27 pictures into one album. That means that Wendy put the remaining 18 pictures into 9 different albums. That means that each album would have 2 pictures. The answer is 2.\n\nQ: For Halloween Katie and her sister combined the candy they received. Katie had 8 pieces of candy while her sister had 23. If they ate 8 pieces the first night, how many pieces do they have left?\nA: Let's think step by step. Katie and her sister have a total of 8 + 23 = 31 pieces of candy. If they eat 8 pieces the first night, they have 31 - 8 = 23 pieces left. The answer is 23.\n\nQ: Bianca was organizing her book case making sure each of the shelves had exactly 8 books on it. If she had 5 shelves of mystery books and 4 shelves of picture books, how many books did she have total?\nA: Let's think step by step. There are 5 shelves of mystery books. Each shelf has 8 books. So that's 40 mystery books. There are 4 shelves of picture books. Each shelf has 8 books. So that's 32 picture books. 40 + 32 = 72 books. The answer is 72.\n\nQ: A pet store had 78 puppies. In one day they sold 30 of them and put the rest into cages with 8 in each cage. How many cages did they use?\nA: Let's think step by step. There are 78 puppies. 30 are sold, so that means there are 48 left. 48 divided by 8 is 6, so that means there are 6 cages with 8 puppies in each. The answer is 6.\n\nQ: A waiter had 14 customers to wait on. If 3 customers left and he got another 39 customers, how many customers would he have?\nA: Let's think step by step. The waiter had 14 customers to wait on. If 3 customers left, that means he would have 11 customers left. If he got another 39 customers, that means he would have 50 customers in total. The answer is 50.\n\nQ: A trivia team had 7 members total, but during a game 2 members didn't show up. If each member that did show up scored 4 points, how many points were scored total?\nA: Let's think step by step. There were 7 members on the team, but 2 members didn't show up. That means that there were 5 members that did show up. Each member that showed up scored 4 points. So if 5 members each scored 4 points, then the total number of points scored would be 5*4=20. The answer is 20.\n\nQ: Gwen had 18 math problems and 11 science problems for homework. If she finished 24 of the problems at school, how many problems did she have to do for homework?\nA: Let's think step by step. Gwen had 18 math problems and 11 science problems for homework. That means she had a total of 29 problems for homework. If she finished 24 of the problems at school, that means she had 5 problems left to do for homework. The answer is 5.\n\nQ: Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left?\nA: Let's think step by step. Mike made $69 from mowing lawns. He spent $24 on new mower blades. That means he has $45 left. Each game costs $5, so he could buy 9 games. The answer is 9.\n\n\nQ: In a video game, each enemy defeated gives you 7 points. If a level has 11 enemies total and you destroy all but 8 of them, how many points would you earn?\nA: Let's think step by step.\n" +} \ No newline at end of file From bf908d22869ed5ca7b92713fe04c26c7d3b3a7c8 Mon Sep 17 00:00:00 2001 From: qinyiwei Date: Sat, 14 Jan 2023 19:36:26 -0500 Subject: [PATCH 3/4] add self_ask --- softwares/react/react.py | 491 ++++++++++++++++------------- softwares/react/self_ask_test.py | 24 ++ softwares/react/software_info.json | 13 +- 3 files changed, 300 insertions(+), 228 deletions(-) create mode 100644 softwares/react/self_ask_test.py diff --git a/softwares/react/react.py b/softwares/react/react.py index 5eac9d0..6ff95bd 100644 --- a/softwares/react/react.py +++ b/softwares/react/react.py @@ -1,48 +1,24 @@ from __future__ import annotations -from promptware.constants.tasks import TaskType +from promptware.tasks import TaskType from promptware.info import SoftwareInfo from promptware.kernels.plm import PLMKernelConfig from promptware.licenses import LicenseType from promptware.promptware import PromptConfig, Promptware import json -import numpy as np import os -import re -import string import json import time -import gym import requests from bs4 import BeautifulSoup +from serpapi import GoogleSearch +from IPython.utils import io - -INSTRUCTION = """Solve a question answering task with interleaving Thought, Action, -Observation steps. Thought can reason about the current situation, and Action can be three types: -(1) Search[entity], which searches the exact entity on Wikipedia and returns the first -paragraph if it exists. If not, it will return some similar entities to search. -(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage. -(3) Finish[answer], which returns the answer and finishes the task. -Here are some examples. -""" - -folder = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) + '/prompts/' -prompt_file = 'prompts_naive.json' -with open(folder + prompt_file, 'r') as f: - PROMPT_DICT = json.load(f) - - -def step(env, action): - attempts = 0 - while attempts < 10: - try: - return env.step(action) - except requests.exceptions.Timeout: - attempts += 1 +serpapi_key = os.getenv("SERPAPI_KEY") -class ReactPromptware(Promptware): +class ChainOfThoughtsPromptware(Promptware): def _info(self) -> SoftwareInfo: return SoftwareInfo( description="This promptware is used to answer a question" @@ -62,13 +38,64 @@ def _kernel_configs(self): platform="openai", model_name="text-davinci-002", temperature=0, - max_tokens=100, + max_tokens=256, top_p=1, frequency_penalty=0.0, presence_penalty=0.0, ) } + def process(self, code): + return code + + def execute(self, input): + kernel = self.kernel_configs["openai"].to_kernel() + code = self.get_code(input, self.default_software_config) + code = self.process(code) + for i in range(1, self.iterations): + kernel.config.stop = [self.get_stop_str(i)] + thought_action = kernel.execute(self.get_execution_code(code, i)) + thought_action = self.normalize_output(thought_action["text"]) + + thought, action = self.get_thought_action(thought_action, kernel, i, code) + + obs, done, answer = self.take_action(action) + obs = obs.replace('\\n', '') + + step_str = self.get_step_str(i, thought, action, obs) + code += step_str + + print(step_str) + if done: + break + + if not done: + obs, done, answer = self.take_final_action(kernel, code, i) + + return answer + + +INSTRUCTION = """Solve a question answering task with interleaving Thought, Action, +Observation steps. Thought can reason about the current situation, and Action can be three types: +(1) Search[entity], which searches the exact entity on Wikipedia and returns the first +paragraph if it exists. If not, it will return some similar entities to search. +(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage. +(3) Finish[answer], which returns the answer and finishes the task. +Here are some examples. +""" + +folder = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) + '/prompts/' +prompt_file = 'prompts_naive.json' +with open(folder + prompt_file, 'r') as f: + PROMPT_DICT = json.load(f) + + +class ReactPromptware(ChainOfThoughtsPromptware): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.env = Wiki() + self.iterations = 8 + def _software_configs(self): return { "multi_hop_qa": PromptConfig( @@ -83,47 +110,65 @@ def _software_configs(self): ) } - def execute(self, input): - env = WikiEnv() - env = HotPotQAWrapper(env) - env = LoggingWrapper(env) + def get_stop_str(self, i): + return f"\nObservation {i}:" - kernel = self.kernel_configs["openai"].to_kernel() - # code += question + "\n" - code = self.get_code(input, self.default_software_config) - env.set(question=input['question']) - n_calls, n_badcalls = 0, 0 - for i in range(1, 8): - n_calls += 1 - kernel.config.stop = [f"\nObservation {i}:"] - thought_action = kernel.execute( - code + f"Thought {i}:") - thought_action = self.normalize_output(thought_action["text"]) - try: - thought, action = thought_action.strip().split(f"\nAction {i}: ") - except: - print('ohh...', thought_action) - n_badcalls += 1 - n_calls += 1 - thought = thought_action.strip().split('\n')[0] - kernel.config['stop'] = [f"\n"] - action = kernel.execute( - code + f"Thought {i}: {thought}\nAction {i}:") - action = self.normalize_output(action["text"]) - obs, r, done, info = step(env, action[0].lower() + action[1:]) - obs = obs.replace('\\n', '') - step_str = f"Thought {i}: {thought}\nAction {i}: {action}\nObservation {i}: {obs}\n" - code += step_str + def get_step_str(self, i, thought, action, obs): + return f"Thought {i}: {thought}\nAction {i}: {action}\nObservation {i}: {obs}\n" - print(step_str) - if done: - break - if not done: - obs, r, done, info = step(env, "finish[]") + def get_execution_code(self, code, i): + return code + f"Thought {i}:" + + def get_thought_action(self, thought_action, kernel, i, code): + try: + thought, action = thought_action.strip().split(f"\nAction {i}: ") + except: + print('ohh...', thought_action) + thought = thought_action.strip().split('\n')[0] + kernel.config['stop'] = [f"\n"] + action = kernel.execute( + code + f"Thought {i}: {thought}\nAction {i}:") + action = self.normalize_output(action["text"]) + + action = (action[0].lower() + action[1:]).strip() + return thought, action + + def take_final_action(self): + self.take_action("finish[]") + + def take_action(self, action): + done = False + answer = None + obs = "" + + if action.startswith("search[") and action.endswith("]"): + entity = action[len("search["):-1] + # entity_ = entity.replace(" ", "_") + # search_url = f"https://en.wikipedia.org/wiki/{entity_}" + obs = self.env.search_step(entity) + elif action.startswith("lookup[") and action.endswith("]"): + keyword = action[len("lookup["):-1] + if self.env.lookup_keyword != keyword: # reset lookup + self.env.lookup_keyword = keyword + self.env.lookup_list = self.env.construct_lookup_list(keyword) + self.env.lookup_cnt = 0 + if self.env.lookup_cnt >= len(self.env.lookup_list): + obs = "No more results.\n" + else: + obs = f"(Result {self.env.lookup_cnt + 1} / {len(self.env.lookup_list)}) " + \ + self.env.lookup_list[self.env.lookup_cnt] + self.env.lookup_cnt += 1 + elif action.startswith("finish[") and action.endswith("]"): + answer = action[len("finish["):-1] + answer = answer + done = True + obs = f"Episode finished\n" + elif action.startswith("think[") and action.endswith("]"): + obs = "Nice thought." + else: + obs = "Invalid action: {}".format(action) - print(info, '\n') - info.update({'n_calls': n_calls, 'n_badcalls': n_badcalls, 'traj': code}) - return info['answer'] + return obs, done, answer def _example(self): return { @@ -136,108 +181,184 @@ def _example(self): } -def normalize_answer(s): - def remove_articles(text): - return re.sub(r"\b(a|an|the)\b", " ", text) +DEMO = ['''Question: Who lived longer, Muhammad Ali or Alan Turing? +Are follow up questions needed here: Yes. +Follow up: How old was Muhammad Ali when he died? +Intermediate answer: Muhammad Ali was 74 years old when he died. +Follow up: How old was Alan Turing when he died? +Intermediate answer: Alan Turing was 41 years old when he died. +So the final answer is: Muhammad Ali + +Question: When was the founder of craigslist born? +Are follow up questions needed here: Yes. +Follow up: Who was the founder of craigslist? +Intermediate answer: Craigslist was founded by Craig Newmark. +Follow up: When was Craig Newmark born? +Intermediate answer: Craig Newmark was born on December 6, 1952. +So the final answer is: December 6, 1952 + +Question: Who was the maternal grandfather of George Washington? +Are follow up questions needed here: Yes. +Follow up: Who was the mother of George Washington? +Intermediate answer: The mother of George Washington was Mary Ball Washington. +Follow up: Who was the father of Mary Ball Washington? +Intermediate answer: The father of Mary Ball Washington was Joseph Ball. +So the final answer is: Joseph Ball + +Question: Are both the directors of Jaws and Casino Royale from the same country? +Are follow up questions needed here: Yes. +Follow up: Who is the director of Jaws? +Intermediate Answer: The director of Jaws is Steven Spielberg. +Follow up: Where is Steven Spielberg from? +Intermediate Answer: The United States. +Follow up: Who is the director of Casino Royale? +Intermediate Answer: The director of Casino Royale is Martin Campbell. +Follow up: Where is Martin Campbell from? +Intermediate Answer: New Zealand. +So the final answer is: No +'''] + + +class SelfAskPromptware(ChainOfThoughtsPromptware): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.iterations = 8 + + def _software_configs(self): + return { + "multi_hop_qa": PromptConfig( + name="multi_hop_qa", + description="This promptware is used to answer a question" + " that involves multiple step reasoning with" + " self ask method.", + instruction="", + demonstration=DEMO, + prompt_template=lambda input: f"Question: {input['question']}\nAre follow up questions needed here:", + task=TaskType.qa_open_domain, + ) + } + + def process(self, code): + return code.strip() + + def get_stop_str(self, i): + return "Intermediate answer:" - def white_space_fix(text): - return " ".join(text.split()) + def get_step_str(self, i, thought, action, obs): + return f"{thought}\nIntermediate answer: {obs}\n" - def remove_punc(text): - exclude = set(string.punctuation) - return "".join(ch for ch in text if ch not in exclude) + def get_execution_code(self, code, i): + return code - def lower(text): - return text.lower() + def get_thought_action(self, thought_action, kernel, i, code): + def get_last_line(generated): + if '\n' not in generated: + last_line = generated + else: + last_line = generated.split('\n')[-1] - return white_space_fix(remove_articles(remove_punc(lower(s)))) + return last_line + def extract_question(generated): + if '\n' not in generated: + last_line = generated + else: + last_line = generated.split('\n')[-1] -class HotPotQAWrapper(gym.Wrapper): - def __init__(self, env): - super().__init__(env) + if 'Follow up:' not in last_line: + print('we probably should never get here...' + generated) - def set(self, question, answer=None): - self.data = (question, answer) + if ':' not in last_line: + after_colon = last_line + else: + after_colon = generated.split(':')[-1] - def _get_info(self): - return { - "steps": self.steps, - "answer": self.answer, - "question": self.data[0], - } + if ' ' == after_colon[0]: + after_colon = after_colon[1:] + if '?' != after_colon[-1]: + print('we probably should never get here...' + generated) + + return after_colon + + if "Follow up:" in get_last_line(thought_action): + action = extract_question(thought_action) + else: + action = extract_question(thought_action) + if "So the final answer is: " in get_last_line(thought_action): + action = "finish[{}]".format(action) + + return thought_action, action + + def take_final_action(self, kernel, code, i): + thought_action = kernel.execute(code+"So the final answer is: ") + answer = self.normalize_output(thought_action["text"]) + obs = answer + done = True + + return obs, done, answer + + def take_action(self, action): + done = False + answer = None + obs = "" - def get_reward(self, info): - if info['answer'] is not None and self.data[1] is not None: - pred = normalize_answer(self.data[1]) - gt = normalize_answer(info['answer']) - score = (pred == gt) - return int(score) - return 0 + if action.startswith("finish[") and action.endswith("]"): + answer = action[len("finish["):-1] + answer = answer + done = True + obs = f"Episode finished\n" + else: + question = action + obs = google(question) - def step(self, action): - # TODO: first step obs does not have question. - obs, _, done, info = self.env.step(action) - reward = self.get_reward(info) - if done: - obs = f"Episode finished, reward = {reward}\n" - info.update( - {"gt_answer": self.data[1]}) - return obs, reward, done, info + return obs, done, answer - def __len__(self): - return len(self.data) + def _example(self): + return { + "input": { + "question": "In what year was the current tallest wooden lattice tower completed?" + }, + "output": 1935, + } -def clean_str(p): - return p.encode().decode("unicode-escape").encode("latin1").decode("utf-8") +def google(question): + params = { + "api_key": serpapi_key, + "engine": "google", + "q": question, + "google_domain": "google.com", + "gl": "us", + "hl": "en" + } + with io.capture_output() as captured: # disables prints from GoogleSearch + search = GoogleSearch(params) + res = search.get_dict() -class textSpace(gym.spaces.Space): - def contains(self, x) -> bool: - """Return boolean specifying if x is a valid member of this space.""" - return isinstance(x, str) + if 'answer_box' in res.keys() and 'answer' in res['answer_box'].keys(): + toret = res['answer_box']['answer'] + elif 'answer_box' in res.keys() and 'snippet' in res['answer_box'].keys(): + toret = res['answer_box']['snippet'] + elif 'answer_box' in res.keys() and 'snippet_highlighted_words' in res['answer_box'].keys(): + toret = res['answer_box']["snippet_highlighted_words"][0] + elif 'snippet' in res["organic_results"][0].keys(): + toret = res["organic_results"][0]['snippet'] + else: + toret = None + return toret -class WikiEnv(gym.Env): +class Wiki(): def __init__(self): - """ - Initialize the environment. - """ - super().__init__() self.page = None # current Wikipedia page - self.obs = None # current observation self.lookup_keyword = None # current lookup keyword self.lookup_list = None # list of paragraphs containing current lookup keyword self.lookup_cnt = None # current lookup index - self.steps = 0 # current number of steps - self.answer = None # current answer from the agent - self.observation_space = self.action_space = textSpace() self.search_time = 0 self.num_searches = 0 - def _get_obs(self): - return self.obs - - def _get_info(self): - return {"steps": self.steps, "answer": self.answer} - - def reset(self, seed=None, return_info=False, options=None): - # We need the following line to seed self.np_random - # super().reset(seed=seed) - self.obs = ("Interact with Wikipedia using search[], lookup[], and " - "finish[].\n") - self.page = None - self.lookup_keyword = None - self.lookup_list = None - self.lookup_cnt = None - self.steps = 0 - self.answer = None - observation = self._get_obs() - info = self._get_info() - return (observation, info) if return_info else observation - def construct_lookup_list(self, keyword): # find all paragraphs if self.page is None: @@ -268,15 +389,6 @@ def get_page_obs(page): sentences = [s.strip() + '.' for s in sentences if s.strip()] return ' '.join(sentences[:5]) - # ps = page.split("\n") - # ret = ps[0] - # for i in range(1, len(ps)): - # if len((ret + ps[i]).split(" ")) <= 50: - # ret += ps[i] - # else: - # break - # return ret - def search_step(self, entity): entity_ = entity.replace(" ", "+") search_url = f"https://en.wikipedia.org/w/index.php?search={entity_}" @@ -287,81 +399,24 @@ def search_step(self, entity): soup = BeautifulSoup(response_text, features="html.parser") result_divs = soup.find_all("div", {"class": "mw-search-result-heading"}) if result_divs: # mismatch - self.result_titles = [clean_str(div.get_text().strip()) + self.result_titles = [self.clean_str(div.get_text().strip()) for div in result_divs] - self.obs = f"Could not find {entity}. Similar: {self.result_titles[:5]}." + obs = f"Could not find {entity}. Similar: {self.result_titles[:5]}." else: page = [p.get_text().strip() for p in soup.find_all("p") + soup.find_all("ul")] if any("may refer to:" in p for p in page): - self.search_step("[" + entity + "]") + obs = self.search_step("[" + entity + "]") else: self.page = "" for p in page: if len(p.split(" ")) > 2: - self.page += clean_str(p) + self.page += self.clean_str(p) if not p.endswith("\n"): self.page += "\n" - self.obs = self.get_page_obs(self.page) + obs = self.get_page_obs(self.page) self.lookup_keyword = self.lookup_list = self.lookup_cnt = None + return obs - def step(self, action): - reward = 0 - done = False - action = action.strip() - if self.answer is not None: # already finished - done = True - return self.obs, reward, done, self._get_info() - - if action.startswith("search[") and action.endswith("]"): - entity = action[len("search["):-1] - # entity_ = entity.replace(" ", "_") - # search_url = f"https://en.wikipedia.org/wiki/{entity_}" - self.search_step(entity) - elif action.startswith("lookup[") and action.endswith("]"): - keyword = action[len("lookup["):-1] - if self.lookup_keyword != keyword: # reset lookup - self.lookup_keyword = keyword - self.lookup_list = self.construct_lookup_list(keyword) - self.lookup_cnt = 0 - if self.lookup_cnt >= len(self.lookup_list): - self.obs = "No more results.\n" - else: - self.obs = f"(Result {self.lookup_cnt + 1} / {len(self.lookup_list)}) " + \ - self.lookup_list[self.lookup_cnt] - self.lookup_cnt += 1 - elif action.startswith("finish[") and action.endswith("]"): - answer = action[len("finish["):-1] - self.answer = answer - done = True - self.obs = f"Episode finished, reward = {reward}\n" - elif action.startswith("think[") and action.endswith("]"): - self.obs = "Nice thought." - else: - self.obs = "Invalid action: {}".format(action) - - self.steps += 1 - - return self.obs, reward, done, self._get_info() - - def get_time_info(self): - speed = self.search_time / self.num_searches if self.num_searches else 0 - return { - "call_speed": speed, - "call_time": self.search_time, - "num_calls": self.num_searches, - } - - -class LoggingWrapper(gym.Wrapper): - def __init__(self, env, folder="trajs", file_id=None): - super().__init__(env) - self.trajs = [] - self.traj = {"observations": [], "actions": []} - self.folder = folder - self.file_id = np.random.randint(0, 10000000) if file_id is None else file_id - self.file_path = f"{self.folder}/{self.file_id}.json" - os.makedirs("trajs", exist_ok=True) - - def __len__(self): - return len(self.env.data) + def clean_str(self, p): + return p.encode().decode("unicode-escape").encode("latin1").decode("utf-8") diff --git a/softwares/react/self_ask_test.py b/softwares/react/self_ask_test.py new file mode 100644 index 0000000..84fe6de --- /dev/null +++ b/softwares/react/self_ask_test.py @@ -0,0 +1,24 @@ +import os +import sys +import unittest + +current_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) +sys.path.append(current_path) + +from react import SelfAskPromptware # noqa + + +class TestSelfAskPromptware(unittest.TestCase): + def test_write_to_directory(self): + software = SelfAskPromptware() + file_path = software.info.write_to_directory(current_path) + self.assertTrue(os.path.exists(file_path)) + + def test_execute_default(self): + software = SelfAskPromptware(config_name="default") + input = { + "question": "In what year was the current tallest wooden lattice tower completed?" + } + result = software.execute(input) + print(result) + self.assertGreater(len(result), 0) diff --git a/softwares/react/software_info.json b/softwares/react/software_info.json index 9545e28..d847c44 100644 --- a/softwares/react/software_info.json +++ b/softwares/react/software_info.json @@ -6,20 +6,13 @@ "codebase_url": "https://github.com/expressai/promptware/tree/main/softwares", "license": "apache-2.0", "task": "qa-open-domain", - "research_tasks": null, - "application_categories": null, - "application_subcategories": null, - "original_platform": null, - "design_pattern": null, - "source_language": null, - "target_language": null, "module_name": "react_promptware", "config_name": "default", "kernel_configs": { "openai": { "platform": "openai", "model_name": "text-davinci-002", - "max_tokens": 100, + "max_tokens": 256, "temperature": 0, "top_p": 1, "suffix": "", @@ -35,7 +28,7 @@ "multi_hop_qa": { "name": "multi_hop_qa", "description": "This promptware is used to answer a question that involves multiple step reasoning with react method.", - "instruction": "Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n", + "instruction": "Solve a question answering task with interleaving Thought, Action, \nObservation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first \nparagraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n", "demonstration": [ "\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n" ], @@ -50,5 +43,5 @@ }, "output": "Beyond the Clouds" }, - "example_prompt": "Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n\n\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n\nQuestion: What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?\n" + "example_prompt": "Solve a question answering task with interleaving Thought, Action, \nObservation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first \nparagraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n\n\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n\nQuestion: What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?\n" } \ No newline at end of file From c5ea2464cd4d34e37b118d9b84e6589223113c90 Mon Sep 17 00:00:00 2001 From: qinyiwei Date: Wed, 1 Feb 2023 19:20:45 -0500 Subject: [PATCH 4/4] Modify React. --- softwares/chain_of_thoughts/__init__.py | 0 softwares/chain_of_thoughts/react/__init__.py | 0 .../react/prompts/prompts_naive.json | 0 .../react/software.py} | 22 +++++++-- .../react/software_info.json | 47 +++++++++++++++++++ .../react/software_test.py} | 18 ++++++- softwares/react/self_ask_test.py | 24 ---------- softwares/react/software_info.json | 47 ------------------- 8 files changed, 81 insertions(+), 77 deletions(-) create mode 100644 softwares/chain_of_thoughts/__init__.py create mode 100644 softwares/chain_of_thoughts/react/__init__.py rename softwares/{ => chain_of_thoughts}/react/prompts/prompts_naive.json (100%) rename softwares/{react/react.py => chain_of_thoughts/react/software.py} (96%) create mode 100644 softwares/chain_of_thoughts/react/software_info.json rename softwares/{react/react_test.py => chain_of_thoughts/react/software_test.py} (55%) delete mode 100644 softwares/react/self_ask_test.py delete mode 100644 softwares/react/software_info.json diff --git a/softwares/chain_of_thoughts/__init__.py b/softwares/chain_of_thoughts/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/softwares/chain_of_thoughts/react/__init__.py b/softwares/chain_of_thoughts/react/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/softwares/react/prompts/prompts_naive.json b/softwares/chain_of_thoughts/react/prompts/prompts_naive.json similarity index 100% rename from softwares/react/prompts/prompts_naive.json rename to softwares/chain_of_thoughts/react/prompts/prompts_naive.json diff --git a/softwares/react/react.py b/softwares/chain_of_thoughts/react/software.py similarity index 96% rename from softwares/react/react.py rename to softwares/chain_of_thoughts/react/software.py index 6ff95bd..3417e0c 100644 --- a/softwares/react/react.py +++ b/softwares/chain_of_thoughts/react/software.py @@ -1,9 +1,17 @@ from __future__ import annotations +from promptware.constants import ( + ApplicationCategory, + ApplicationSubcategory, + DesignPatternType, + LanguageType, + LicenseType, + PlatformType, + TaskType, +) from promptware.tasks import TaskType from promptware.info import SoftwareInfo from promptware.kernels.plm import PLMKernelConfig -from promptware.licenses import LicenseType from promptware.promptware import PromptConfig, Promptware import json @@ -13,7 +21,6 @@ import requests from bs4 import BeautifulSoup from serpapi import GoogleSearch -from IPython.utils import io serpapi_key = os.getenv("SERPAPI_KEY") @@ -30,6 +37,12 @@ def _info(self) -> SoftwareInfo: codebase_url="https://github.com/expressai/promptware/tree/main/softwares", license=LicenseType.apache_2_0, task=TaskType.qa_open_domain, + application_categories=[ApplicationCategory.conversation], + application_subcategories=[ApplicationSubcategory.question_answering], + original_platform=PlatformType.gpt3, + design_pattern=DesignPatternType.chain_of_thought_action, + source_language=LanguageType.en, + target_language=LanguageType.en, ) def _kernel_configs(self): @@ -332,9 +345,8 @@ def google(question): "hl": "en" } - with io.capture_output() as captured: # disables prints from GoogleSearch - search = GoogleSearch(params) - res = search.get_dict() + search = GoogleSearch(params) + res = search.get_dict() if 'answer_box' in res.keys() and 'answer' in res['answer_box'].keys(): toret = res['answer_box']['answer'] diff --git a/softwares/chain_of_thoughts/react/software_info.json b/softwares/chain_of_thoughts/react/software_info.json new file mode 100644 index 0000000..1ced075 --- /dev/null +++ b/softwares/chain_of_thoughts/react/software_info.json @@ -0,0 +1,47 @@ +{ + "description": "This promptware is used to answer a question that involves multiple step reasoning with chain-of-thought method.", + "creator": "Promptware Authors", + "homepage": "https://github.com/expressai/promptware", + "reference": "", + "codebase_url": "https://github.com/expressai/promptware/tree/main/softwares", + "license": "apache-2.0", + "task": "qa-open-domain", + "module_name": "self_ask_promptware", + "config_name": "default", + "kernel_configs": { + "openai": { + "platform": "openai", + "model_name": "text-davinci-002", + "max_tokens": 256, + "temperature": 0, + "top_p": 1, + "suffix": "", + "frequency_penalty": 0.0, + "presence_penalty": 0.0, + "logprobs": 1, + "n": 1, + "echo": false, + "stop": null + } + }, + "software_configs": { + "multi_hop_qa": { + "name": "multi_hop_qa", + "description": "This promptware is used to answer a question that involves multiple step reasoning with self ask method.", + "instruction": "", + "demonstration": [ + "Question: Who lived longer, Muhammad Ali or Alan Turing?\nAre follow up questions needed here: Yes.\nFollow up: How old was Muhammad Ali when he died?\nIntermediate answer: Muhammad Ali was 74 years old when he died.\nFollow up: How old was Alan Turing when he died?\nIntermediate answer: Alan Turing was 41 years old when he died.\nSo the final answer is: Muhammad Ali \n\nQuestion: When was the founder of craigslist born?\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of craigslist?\nIntermediate answer: Craigslist was founded by Craig Newmark.\nFollow up: When was Craig Newmark born?\nIntermediate answer: Craig Newmark was born on December 6, 1952.\nSo the final answer is: December 6, 1952\n\nQuestion: Who was the maternal grandfather of George Washington?\nAre follow up questions needed here: Yes.\nFollow up: Who was the mother of George Washington?\nIntermediate answer: The mother of George Washington was Mary Ball Washington.\nFollow up: Who was the father of Mary Ball Washington?\nIntermediate answer: The father of Mary Ball Washington was Joseph Ball.\nSo the final answer is: Joseph Ball \n\nQuestion: Are both the directors of Jaws and Casino Royale from the same country? \nAre follow up questions needed here: Yes. \nFollow up: Who is the director of Jaws? \nIntermediate Answer: The director of Jaws is Steven Spielberg. \nFollow up: Where is Steven Spielberg from? \nIntermediate Answer: The United States. \nFollow up: Who is the director of Casino Royale? \nIntermediate Answer: The director of Casino Royale is Martin Campbell. \nFollow up: Where is Martin Campbell from? \nIntermediate Answer: New Zealand. \nSo the final answer is: No\n" + ], + "prompt_template": "lambda input: f\"Question: {input['question']}\\nAre follow up questions needed here:\",", + "task": "qa-open-domain" + } + }, + "dataset_configs": null, + "example": { + "input": { + "question": "In what year was the current tallest wooden lattice tower completed?" + }, + "output": 1935 + }, + "example_prompt": "Question: Who lived longer, Muhammad Ali or Alan Turing?\nAre follow up questions needed here: Yes.\nFollow up: How old was Muhammad Ali when he died?\nIntermediate answer: Muhammad Ali was 74 years old when he died.\nFollow up: How old was Alan Turing when he died?\nIntermediate answer: Alan Turing was 41 years old when he died.\nSo the final answer is: Muhammad Ali \n\nQuestion: When was the founder of craigslist born?\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of craigslist?\nIntermediate answer: Craigslist was founded by Craig Newmark.\nFollow up: When was Craig Newmark born?\nIntermediate answer: Craig Newmark was born on December 6, 1952.\nSo the final answer is: December 6, 1952\n\nQuestion: Who was the maternal grandfather of George Washington?\nAre follow up questions needed here: Yes.\nFollow up: Who was the mother of George Washington?\nIntermediate answer: The mother of George Washington was Mary Ball Washington.\nFollow up: Who was the father of Mary Ball Washington?\nIntermediate answer: The father of Mary Ball Washington was Joseph Ball.\nSo the final answer is: Joseph Ball \n\nQuestion: Are both the directors of Jaws and Casino Royale from the same country? \nAre follow up questions needed here: Yes. \nFollow up: Who is the director of Jaws? \nIntermediate Answer: The director of Jaws is Steven Spielberg. \nFollow up: Where is Steven Spielberg from? \nIntermediate Answer: The United States. \nFollow up: Who is the director of Casino Royale? \nIntermediate Answer: The director of Casino Royale is Martin Campbell. \nFollow up: Where is Martin Campbell from? \nIntermediate Answer: New Zealand. \nSo the final answer is: No\n\nQuestion: In what year was the current tallest wooden lattice tower completed?\nAre follow up questions needed here:\n" +} \ No newline at end of file diff --git a/softwares/react/react_test.py b/softwares/chain_of_thoughts/react/software_test.py similarity index 55% rename from softwares/react/react_test.py rename to softwares/chain_of_thoughts/react/software_test.py index 4afdd81..b4aa42c 100644 --- a/softwares/react/react_test.py +++ b/softwares/chain_of_thoughts/react/software_test.py @@ -5,7 +5,7 @@ current_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) sys.path.append(current_path) -from react import ReactPromptware # noqa +from .software import ReactPromptware, SelfAskPromptware # noqa class TestReactPromptware(unittest.TestCase): @@ -24,3 +24,19 @@ def test_execute_default(self): result = software.execute(input) print(result) self.assertGreater(len(result), 0) + + +class TestSelfAskPromptware(unittest.TestCase): + def test_write_to_directory(self): + software = SelfAskPromptware() + file_path = software.info.write_to_directory(current_path) + self.assertTrue(os.path.exists(file_path)) + + def test_execute_default(self): + software = SelfAskPromptware(config_name="default") + input = { + "question": "In what year was the current tallest wooden lattice tower completed?" + } + result = software.execute(input) + print(result) + self.assertGreater(len(result), 0) diff --git a/softwares/react/self_ask_test.py b/softwares/react/self_ask_test.py deleted file mode 100644 index 84fe6de..0000000 --- a/softwares/react/self_ask_test.py +++ /dev/null @@ -1,24 +0,0 @@ -import os -import sys -import unittest - -current_path = os.path.abspath(os.path.join(os.path.dirname(__file__), ".")) -sys.path.append(current_path) - -from react import SelfAskPromptware # noqa - - -class TestSelfAskPromptware(unittest.TestCase): - def test_write_to_directory(self): - software = SelfAskPromptware() - file_path = software.info.write_to_directory(current_path) - self.assertTrue(os.path.exists(file_path)) - - def test_execute_default(self): - software = SelfAskPromptware(config_name="default") - input = { - "question": "In what year was the current tallest wooden lattice tower completed?" - } - result = software.execute(input) - print(result) - self.assertGreater(len(result), 0) diff --git a/softwares/react/software_info.json b/softwares/react/software_info.json deleted file mode 100644 index d847c44..0000000 --- a/softwares/react/software_info.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "description": "This promptware is used to answer a question that involves multiple step reasoning with chain-of-thought method.", - "creator": "Promptware Authors", - "homepage": "https://github.com/expressai/promptware", - "reference": "", - "codebase_url": "https://github.com/expressai/promptware/tree/main/softwares", - "license": "apache-2.0", - "task": "qa-open-domain", - "module_name": "react_promptware", - "config_name": "default", - "kernel_configs": { - "openai": { - "platform": "openai", - "model_name": "text-davinci-002", - "max_tokens": 256, - "temperature": 0, - "top_p": 1, - "suffix": "", - "frequency_penalty": 0.0, - "presence_penalty": 0.0, - "logprobs": 1, - "n": 1, - "echo": false, - "stop": null - } - }, - "software_configs": { - "multi_hop_qa": { - "name": "multi_hop_qa", - "description": "This promptware is used to answer a question that involves multiple step reasoning with react method.", - "instruction": "Solve a question answering task with interleaving Thought, Action, \nObservation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first \nparagraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n", - "demonstration": [ - "\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n" - ], - "prompt_template": "lambda input: f\"Question: {input['question']}\",", - "task": "qa-open-domain" - } - }, - "dataset_configs": null, - "example": { - "input": { - "question": "What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?" - }, - "output": "Beyond the Clouds" - }, - "example_prompt": "Solve a question answering task with interleaving Thought, Action, \nObservation steps. Thought can reason about the current situation, and Action can be three types: \n(1) Search[entity], which searches the exact entity on Wikipedia and returns the first \nparagraph if it exists. If not, it will return some similar entities to search.\n(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.\n(3) Finish[answer], which returns the answer and finishes the task.\nHere are some examples.\n\n\nQuestion: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?\nThought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.\nAction 1: Search[Colorado orogeny]\nObservation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas.\nThought 2: It does not mention the eastern sector. So I need to look up eastern sector.\nAction 2: Lookup[eastern sector]\nObservation 2: (Result 1 / 1) The eastern sector extends into the High Plains and is called the Central Plains orogeny.\nThought 3: The eastern sector of Colorado orogeny extends into the High Plains. So I need to search High Plains and find its elevation range.\nAction 3: Search[High Plains]\nObservation 3: High Plains refers to one of two distinct land regions:\nThought 4: I need to instead search High Plains (United States).\nAction 4: Search[High Plains (United States)]\nObservation 4: The High Plains are a subregion of the Great Plains. From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]\nThought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.\nAction 5: Finish[1,800 to 7,000 ft]\nQuestion: Musician and satirist Allie Goertz wrote a song about the \"The Simpsons\" character Milhouse, who Matt Groening named after who?\nThought 1: The question simplifies to \"The Simpsons\" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.\nAction 1: Search[Milhouse]\nObservation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.\nThought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up \"named after\".\nAction 2: Lookup[named after]\nObservation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous. \nThought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.\nAction 3: Finish[Richard Nixon]\nQuestion: Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?\nThought 1: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which documentary is about Finnish rock groups.\nAction 1: Search[Adam Clayton Powell]\nObservation 1: Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office Building', 'Isabel Washington Powell', 'Adam Powell', 'Adam Clayton Powell (film)', 'Giancarlo Esposito'].\nThought 2: To find the documentary, I can search Adam Clayton Powell (film).\nAction 2: Search[Adam Clayton Powell (film)]\nObservation 2: Adam Clayton Powell is a 1989 American documentary film directed by Richard Kilberg.\nThe film is about the rise and fall of influential African-American politician Adam Clayton Powell Jr.[3][4] It was later aired as part of the PBS series The American Experience.\nThought 3: Adam Clayton Powell (film) is a documentary about an African-American politician, not Finnish rock groups. So the documentary about Finnish rock groups must instead be The Saimaa Gesture.\nAction 3: Finish[The Saimaa Gesture]\nQuestion: What profession does Nicholas Ray and Elia Kazan have in common?\nThought 1: I need to search Nicholas Ray and Elia Kazan, find their professions, then find the profession they have in common.\nAction 1: Search[Nicholas Ray]\nObservation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 \u2013 June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.\nThought 2: Professions of Nicholas Ray are director, screenwriter, and actor. I need to search Elia Kazan next and find his professions.\nAction 2: Search[Elia Kazan]\nObservation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.\nThought 3: Professions of Elia Kazan are director, producer, screenwriter, and actor. So profession Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.\nAction 3: Finish[director, screenwriter, actor]\nQuestion: Which magazine was started first Arthur's Magazine or First for Women?\nThought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.\nAction 1: Search[Arthur's Magazine]\nObservation 1: Arthur's Magazine (1844-\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. \nThought 2: Arthur's Magazine was started in 1844. I need to search First for Women next.\nAction 2: Search[First for Women]\nObservation 2: First for Women is a woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. \nThought 3: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.\nAction 3: Finish[Arthur's Magazine]\nQuestion: Were Pavel Urysohn and Leonid Levin known for the same type of work?\nThought 1: I need to search Pavel Urysohn and Leonid Levin, find their types of work, then find if they are the same.\nAction 1: Search[Pavel Urysohn]\nObservation 1: Pavel Samuilovich Urysohn (February 3, 1898 \u00e2\u0080\u0093 August 17, 1924) was a Soviet mathematician who is best known for his contributions in dimension theory.\nThought 2: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and find its type of work.\nAction 2: Search[Leonid Levin]\nObservation 2: Leonid Anatolievich Levin is a Soviet-American mathematician and computer scientist. \nThought 3: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn and Leonid Levin have the same type of work. \nAction 3: Finish[yes]\n\nQuestion: What movie did actress Irene Jacob complete before the American action crime thriller film directed by Stuart Bird?\n" -} \ No newline at end of file